Note: This is test shard 2 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from RebalanceIgnoredTserversTest
[ RUN ] RebalanceIgnoredTserversTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 13:58:35.878897 398 test_util.cc:268] Using random seed: 497787357
I20240503 13:58:35.886426 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 13:58:35.886502 398 ts_itest-base.cc:113] --------------
I20240503 13:58:35.886519 398 ts_itest-base.cc:114] 5 tablet servers
I20240503 13:58:35.886538 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 13:58:35.886556 398 ts_itest-base.cc:116] --------------
2024-05-03T13:58:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T13:58:35Z Disabled control of system clock
I20240503 13:58:35.896062 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:42995
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:34243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:42995
--enable_range_replica_placement=false with env {}
W20240503 13:58:36.010644 410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:36.010921 410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:36.010998 410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:36.014370 410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 13:58:36.014456 410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:36.014489 410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 13:58:36.014535 410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 13:58:36.018520 410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:42995
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:42995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:36.019594 410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:36.020833 410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:36.029316 415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:36.029912 418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:36.030581 416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:36.031601 410 server_base.cc:1034] running on GCE node
I20240503 13:58:36.032204 410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:36.033214 410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:36.034361 410 hybrid_clock.cc:648] HybridClock initialized: now 1714744716034303 us; error 78 us; skew 500 ppm
I20240503 13:58:36.036410 410 webserver.cc:460] Webserver started at http://127.0.99.190:45287/ using document root <none> and password file <none>
I20240503 13:58:36.037072 410 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:36.037158 410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:36.037375 410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:36.039093 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "47b8e5cf7868441db19571d1a9835d0e"
format_stamp: "Formatted at 2024-05-03 13:58:36 on dist-test-slave-fr9h"
I20240503 13:58:36.039665 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "47b8e5cf7868441db19571d1a9835d0e"
format_stamp: "Formatted at 2024-05-03 13:58:36 on dist-test-slave-fr9h"
I20240503 13:58:36.043071 410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 13:58:36.045622 424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:36.046672 410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:36.046798 410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "47b8e5cf7868441db19571d1a9835d0e"
format_stamp: "Formatted at 2024-05-03 13:58:36 on dist-test-slave-fr9h"
I20240503 13:58:36.046888 410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:36.058840 410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:36.059551 410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:36.059722 410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:36.066555 476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:42995 every 8 connection(s)
I20240503 13:58:36.066568 410 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:42995
I20240503 13:58:36.067816 410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 13:58:36.070387 477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:36.072017 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 410
I20240503 13:58:36.072158 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 13:58:36.076503 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47b8e5cf7868441db19571d1a9835d0e: Bootstrap starting.
I20240503 13:58:36.079089 477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47b8e5cf7868441db19571d1a9835d0e: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:36.079891 477 log.cc:824] T 00000000000000000000000000000000 P 47b8e5cf7868441db19571d1a9835d0e: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:36.081812 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47b8e5cf7868441db19571d1a9835d0e: No bootstrap required, opened a new log
I20240503 13:58:36.084926 477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47b8e5cf7868441db19571d1a9835d0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47b8e5cf7868441db19571d1a9835d0e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42995 } }
I20240503 13:58:36.085143 477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47b8e5cf7868441db19571d1a9835d0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:36.085259 477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47b8e5cf7868441db19571d1a9835d0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47b8e5cf7868441db19571d1a9835d0e, State: Initialized, Role: FOLLOWER
I20240503 13:58:36.085832 477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47b8e5cf7868441db19571d1a9835d0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47b8e5cf7868441db19571d1a9835d0e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42995 } }
I20240503 13:58:36.086032 477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47b8e5cf7868441db19571d1a9835d0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:58:36.086128 477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47b8e5cf7868441db19571d1a9835d0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:58:36.086256 477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47b8e5cf7868441db19571d1a9835d0e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:36.126924 477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47b8e5cf7868441db19571d1a9835d0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47b8e5cf7868441db19571d1a9835d0e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42995 } }
I20240503 13:58:36.127501 477 leader_election.cc:302] T 00000000000000000000000000000000 P 47b8e5cf7868441db19571d1a9835d0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47b8e5cf7868441db19571d1a9835d0e; no voters:
I20240503 13:58:36.127918 477 leader_election.cc:288] T 00000000000000000000000000000000 P 47b8e5cf7868441db19571d1a9835d0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:58:36.128300 482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47b8e5cf7868441db19571d1a9835d0e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:36.128566 482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47b8e5cf7868441db19571d1a9835d0e [term 1 LEADER]: Becoming Leader. State: Replica: 47b8e5cf7868441db19571d1a9835d0e, State: Running, Role: LEADER
I20240503 13:58:36.129020 482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47b8e5cf7868441db19571d1a9835d0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47b8e5cf7868441db19571d1a9835d0e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42995 } }
I20240503 13:58:36.129153 477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47b8e5cf7868441db19571d1a9835d0e [sys.catalog]: configured and running, proceeding with master startup.
I20240503 13:58:36.130739 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47b8e5cf7868441db19571d1a9835d0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47b8e5cf7868441db19571d1a9835d0e. Latest consensus state: current_term: 1 leader_uuid: "47b8e5cf7868441db19571d1a9835d0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47b8e5cf7868441db19571d1a9835d0e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42995 } } }
I20240503 13:58:36.130793 483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47b8e5cf7868441db19571d1a9835d0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47b8e5cf7868441db19571d1a9835d0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47b8e5cf7868441db19571d1a9835d0e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42995 } } }
I20240503 13:58:36.130877 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47b8e5cf7868441db19571d1a9835d0e [sys.catalog]: This master's current role is: LEADER
I20240503 13:58:36.130901 483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47b8e5cf7868441db19571d1a9835d0e [sys.catalog]: This master's current role is: LEADER
I20240503 13:58:36.131757 493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 13:58:36.134367 493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 13:58:36.140199 493 catalog_manager.cc:1347] Generated new cluster ID: b1d667af79264d818e36eb761a7f4e82
I20240503 13:58:36.140278 493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 13:58:36.154634 493 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 13:58:36.155603 493 catalog_manager.cc:1504] Loading token signing keys...
I20240503 13:58:36.164640 493 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 47b8e5cf7868441db19571d1a9835d0e: Generated new TSK 0
I20240503 13:58:36.165334 493 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 13:58:36.179742 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42995
--builtin_ntp_servers=127.0.99.148:34243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240503 13:58:36.295058 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:36.295262 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:36.295358 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:36.298440 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:36.298556 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 13:58:36.302239 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42995
--never_fsync=true
--heap_profile_path=/tmp/kudu.501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:36.303126 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:36.304291 501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:36.312849 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:36.313529 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:36.314357 501 server_base.cc:1034] running on GCE node
W20240503 13:58:36.315014 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:36.315521 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:36.316097 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:36.317245 501 hybrid_clock.cc:648] HybridClock initialized: now 1714744716317225 us; error 39 us; skew 500 ppm
I20240503 13:58:36.319171 501 webserver.cc:460] Webserver started at http://127.0.99.129:42123/ using document root <none> and password file <none>
I20240503 13:58:36.319797 501 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:36.319857 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:36.320042 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:36.321760 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "af52d7f74d294bc8828d7ffec9cd3ca4"
format_stamp: "Formatted at 2024-05-03 13:58:36 on dist-test-slave-fr9h"
I20240503 13:58:36.322252 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "af52d7f74d294bc8828d7ffec9cd3ca4"
format_stamp: "Formatted at 2024-05-03 13:58:36 on dist-test-slave-fr9h"
I20240503 13:58:36.325891 501 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240503 13:58:36.328481 515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:36.329767 501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:36.329910 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "af52d7f74d294bc8828d7ffec9cd3ca4"
format_stamp: "Formatted at 2024-05-03 13:58:36 on dist-test-slave-fr9h"
I20240503 13:58:36.330017 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:36.346989 501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:36.347734 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:36.347904 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:36.348484 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:36.349407 501 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:36.349459 501 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:36.349519 501 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:36.349546 501 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:36.358745 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:39031
I20240503 13:58:36.358764 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:39031 every 8 connection(s)
I20240503 13:58:36.359949 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 13:58:36.365792 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 501
I20240503 13:58:36.365904 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 13:58:36.368252 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42995
--builtin_ntp_servers=127.0.99.148:34243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 13:58:36.369511 629 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42995
I20240503 13:58:36.369800 629 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:36.370350 629 heartbeater.cc:511] Master 127.0.99.190:42995 requested a full tablet report, sending...
I20240503 13:58:36.371953 441 ts_manager.cc:194] Registered new tserver with Master: af52d7f74d294bc8828d7ffec9cd3ca4 (127.0.99.129:39031)
I20240503 13:58:36.373193 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:35309
W20240503 13:58:36.480501 632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:36.480747 632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:36.480823 632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:36.483979 632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:36.484131 632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 13:58:36.487871 632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42995
--never_fsync=true
--heap_profile_path=/tmp/kudu.632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:36.488782 632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:36.489924 632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:36.498740 637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:36.499425 638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:36.500115 640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:36.500912 632 server_base.cc:1034] running on GCE node
I20240503 13:58:36.501286 632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:36.501888 632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:36.503062 632 hybrid_clock.cc:648] HybridClock initialized: now 1714744716503050 us; error 26 us; skew 500 ppm
I20240503 13:58:36.505007 632 webserver.cc:460] Webserver started at http://127.0.99.130:35885/ using document root <none> and password file <none>
I20240503 13:58:36.505605 632 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:36.505679 632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:36.505889 632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:36.507787 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "c72fe65a1c474fa1b0670828e00f6d27"
format_stamp: "Formatted at 2024-05-03 13:58:36 on dist-test-slave-fr9h"
I20240503 13:58:36.508283 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "c72fe65a1c474fa1b0670828e00f6d27"
format_stamp: "Formatted at 2024-05-03 13:58:36 on dist-test-slave-fr9h"
I20240503 13:58:36.511695 632 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 13:58:36.513878 646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:36.515036 632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:36.515157 632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c72fe65a1c474fa1b0670828e00f6d27"
format_stamp: "Formatted at 2024-05-03 13:58:36 on dist-test-slave-fr9h"
I20240503 13:58:36.515260 632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:36.535876 632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:36.536549 632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:36.536722 632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:36.537279 632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:36.538203 632 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:36.538255 632 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:36.538316 632 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:36.538352 632 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:36.549541 632 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:32785
I20240503 13:58:36.549574 759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:32785 every 8 connection(s)
I20240503 13:58:36.550544 632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 13:58:36.554623 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 632
I20240503 13:58:36.554733 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 13:58:36.557081 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42995
--builtin_ntp_servers=127.0.99.148:34243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 13:58:36.559847 760 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42995
I20240503 13:58:36.560091 760 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:36.560617 760 heartbeater.cc:511] Master 127.0.99.190:42995 requested a full tablet report, sending...
I20240503 13:58:36.561689 441 ts_manager.cc:194] Registered new tserver with Master: c72fe65a1c474fa1b0670828e00f6d27 (127.0.99.130:32785)
I20240503 13:58:36.562285 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:36161
W20240503 13:58:36.675559 763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:36.675786 763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:36.675830 763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:36.678833 763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:36.678934 763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 13:58:36.682582 763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42995
--never_fsync=true
--heap_profile_path=/tmp/kudu.763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:36.683490 763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:36.684638 763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:36.693353 769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:36.694052 768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:36.694818 771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:36.695560 763 server_base.cc:1034] running on GCE node
I20240503 13:58:36.695916 763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:36.696477 763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:36.697674 763 hybrid_clock.cc:648] HybridClock initialized: now 1714744716697660 us; error 24 us; skew 500 ppm
I20240503 13:58:36.699486 763 webserver.cc:460] Webserver started at http://127.0.99.131:44037/ using document root <none> and password file <none>
I20240503 13:58:36.700058 763 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:36.700117 763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:36.700284 763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:36.702095 763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e16f7d9da90c4521a8e763bca0084376"
format_stamp: "Formatted at 2024-05-03 13:58:36 on dist-test-slave-fr9h"
I20240503 13:58:36.702713 763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e16f7d9da90c4521a8e763bca0084376"
format_stamp: "Formatted at 2024-05-03 13:58:36 on dist-test-slave-fr9h"
I20240503 13:58:36.706166 763 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240503 13:58:36.708441 777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:36.709463 763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:36.709561 763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e16f7d9da90c4521a8e763bca0084376"
format_stamp: "Formatted at 2024-05-03 13:58:36 on dist-test-slave-fr9h"
I20240503 13:58:36.709650 763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:36.743912 763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:36.744625 763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:36.744815 763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:36.745398 763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:36.746423 763 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:36.746484 763 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:36.746531 763 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:36.746568 763 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:36.756815 763 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:44007
I20240503 13:58:36.756853 890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:44007 every 8 connection(s)
I20240503 13:58:36.757951 763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 13:58:36.763511 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 763
I20240503 13:58:36.763614 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 13:58:36.765895 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42995
--builtin_ntp_servers=127.0.99.148:34243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 13:58:36.767634 891 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42995
I20240503 13:58:36.767911 891 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:36.768637 891 heartbeater.cc:511] Master 127.0.99.190:42995 requested a full tablet report, sending...
I20240503 13:58:36.770696 441 ts_manager.cc:194] Registered new tserver with Master: e16f7d9da90c4521a8e763bca0084376 (127.0.99.131:44007)
I20240503 13:58:36.771574 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:41111
W20240503 13:58:36.891364 894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:36.891597 894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:36.891665 894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:36.894757 894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:36.894896 894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 13:58:36.898671 894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42995
--never_fsync=true
--heap_profile_path=/tmp/kudu.894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:36.899633 894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:36.900748 894 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:36.910051 900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:36.910931 902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:36.911741 899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:36.912528 894 server_base.cc:1034] running on GCE node
I20240503 13:58:36.912961 894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:36.913596 894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:36.914783 894 hybrid_clock.cc:648] HybridClock initialized: now 1714744716914769 us; error 32 us; skew 500 ppm
I20240503 13:58:36.916775 894 webserver.cc:460] Webserver started at http://127.0.99.132:41715/ using document root <none> and password file <none>
I20240503 13:58:36.917371 894 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:36.917433 894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:36.917608 894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:36.919412 894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "45bbe6ca5c69471cbfaa002a655bdcd9"
format_stamp: "Formatted at 2024-05-03 13:58:36 on dist-test-slave-fr9h"
I20240503 13:58:36.920060 894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "45bbe6ca5c69471cbfaa002a655bdcd9"
format_stamp: "Formatted at 2024-05-03 13:58:36 on dist-test-slave-fr9h"
I20240503 13:58:36.925444 894 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20240503 13:58:36.927743 908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:36.928876 894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:58:36.928973 894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "45bbe6ca5c69471cbfaa002a655bdcd9"
format_stamp: "Formatted at 2024-05-03 13:58:36 on dist-test-slave-fr9h"
I20240503 13:58:36.929093 894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:36.955734 894 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:36.956369 894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:36.956514 894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:36.957043 894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:36.957906 894 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:36.957952 894 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:36.957991 894 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:36.958009 894 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:36.968827 894 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:38173
I20240503 13:58:36.968863 1021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:38173 every 8 connection(s)
I20240503 13:58:36.969825 894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 13:58:36.973085 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 894
I20240503 13:58:36.973263 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 13:58:36.975603 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42995
--builtin_ntp_servers=127.0.99.148:34243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 13:58:36.979770 1022 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42995
I20240503 13:58:36.979993 1022 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:36.980588 1022 heartbeater.cc:511] Master 127.0.99.190:42995 requested a full tablet report, sending...
I20240503 13:58:36.981724 441 ts_manager.cc:194] Registered new tserver with Master: 45bbe6ca5c69471cbfaa002a655bdcd9 (127.0.99.132:38173)
I20240503 13:58:36.982278 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:52173
W20240503 13:58:37.096607 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:37.096833 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:37.096879 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:37.100119 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:37.100232 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 13:58:37.104435 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42995
--never_fsync=true
--heap_profile_path=/tmp/kudu.1025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:37.105384 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:37.106542 1025 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:37.115530 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:37.116247 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:37.117482 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:37.117878 1025 server_base.cc:1034] running on GCE node
I20240503 13:58:37.118312 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:37.118978 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:37.120144 1025 hybrid_clock.cc:648] HybridClock initialized: now 1714744717120117 us; error 38 us; skew 500 ppm
I20240503 13:58:37.122108 1025 webserver.cc:460] Webserver started at http://127.0.99.133:40513/ using document root <none> and password file <none>
I20240503 13:58:37.122664 1025 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:37.122733 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:37.122910 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:37.124722 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "892780a68fd9443ebad27d673d6d0d23"
format_stamp: "Formatted at 2024-05-03 13:58:37 on dist-test-slave-fr9h"
I20240503 13:58:37.125229 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "892780a68fd9443ebad27d673d6d0d23"
format_stamp: "Formatted at 2024-05-03 13:58:37 on dist-test-slave-fr9h"
I20240503 13:58:37.128981 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240503 13:58:37.131374 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:37.132375 1025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:37.132472 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "892780a68fd9443ebad27d673d6d0d23"
format_stamp: "Formatted at 2024-05-03 13:58:37 on dist-test-slave-fr9h"
I20240503 13:58:37.132563 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:37.144516 1025 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:37.145108 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:37.145265 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:37.145777 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:37.146623 1025 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:37.146672 1025 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:37.146739 1025 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:37.146776 1025 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:37.157245 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:36003
I20240503 13:58:37.157281 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:36003 every 8 connection(s)
I20240503 13:58:37.158164 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 13:58:37.161527 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 1025
I20240503 13:58:37.161640 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 13:58:37.168454 1153 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42995
I20240503 13:58:37.168650 1153 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:37.169139 1153 heartbeater.cc:511] Master 127.0.99.190:42995 requested a full tablet report, sending...
I20240503 13:58:37.170006 441 ts_manager.cc:194] Registered new tserver with Master: 892780a68fd9443ebad27d673d6d0d23 (127.0.99.133:36003)
I20240503 13:58:37.170647 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:53453
I20240503 13:58:37.176266 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 13:58:37.189167 441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44320:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 13:58:37.207880 1085 tablet_service.cc:1467] Processing CreateTablet for tablet 84c31335aee24600945210f193348807 (DEFAULT_TABLE table=TestTable [id=13101edcc3bf4c30b227146df0866b3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:37.207924 563 tablet_service.cc:1467] Processing CreateTablet for tablet 84c31335aee24600945210f193348807 (DEFAULT_TABLE table=TestTable [id=13101edcc3bf4c30b227146df0866b3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:37.207911 823 tablet_service.cc:1467] Processing CreateTablet for tablet 84c31335aee24600945210f193348807 (DEFAULT_TABLE table=TestTable [id=13101edcc3bf4c30b227146df0866b3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:37.208851 1085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84c31335aee24600945210f193348807. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:37.208925 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84c31335aee24600945210f193348807. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:37.208925 823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84c31335aee24600945210f193348807. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:37.214231 1171 tablet_bootstrap.cc:492] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23: Bootstrap starting.
I20240503 13:58:37.215488 1172 tablet_bootstrap.cc:492] T 84c31335aee24600945210f193348807 P af52d7f74d294bc8828d7ffec9cd3ca4: Bootstrap starting.
I20240503 13:58:37.216871 1171 tablet_bootstrap.cc:654] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:37.217411 1172 tablet_bootstrap.cc:654] T 84c31335aee24600945210f193348807 P af52d7f74d294bc8828d7ffec9cd3ca4: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:37.217486 1171 log.cc:824] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:37.218041 1173 tablet_bootstrap.cc:492] T 84c31335aee24600945210f193348807 P e16f7d9da90c4521a8e763bca0084376: Bootstrap starting.
I20240503 13:58:37.218127 1172 log.cc:824] T 84c31335aee24600945210f193348807 P af52d7f74d294bc8828d7ffec9cd3ca4: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:37.219310 1171 tablet_bootstrap.cc:492] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23: No bootstrap required, opened a new log
I20240503 13:58:37.219548 1171 ts_tablet_manager.cc:1367] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20240503 13:58:37.219859 1173 tablet_bootstrap.cc:654] T 84c31335aee24600945210f193348807 P e16f7d9da90c4521a8e763bca0084376: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:37.219899 1172 tablet_bootstrap.cc:492] T 84c31335aee24600945210f193348807 P af52d7f74d294bc8828d7ffec9cd3ca4: No bootstrap required, opened a new log
I20240503 13:58:37.220037 1172 ts_tablet_manager.cc:1367] T 84c31335aee24600945210f193348807 P af52d7f74d294bc8828d7ffec9cd3ca4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240503 13:58:37.220477 1173 log.cc:824] T 84c31335aee24600945210f193348807 P e16f7d9da90c4521a8e763bca0084376: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:37.221956 1173 tablet_bootstrap.cc:492] T 84c31335aee24600945210f193348807 P e16f7d9da90c4521a8e763bca0084376: No bootstrap required, opened a new log
I20240503 13:58:37.222167 1173 ts_tablet_manager.cc:1367] T 84c31335aee24600945210f193348807 P e16f7d9da90c4521a8e763bca0084376: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 13:58:37.223632 1171 raft_consensus.cc:348] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } }
I20240503 13:58:37.223632 1172 raft_consensus.cc:348] T 84c31335aee24600945210f193348807 P af52d7f74d294bc8828d7ffec9cd3ca4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } }
I20240503 13:58:37.223858 1172 raft_consensus.cc:374] T 84c31335aee24600945210f193348807 P af52d7f74d294bc8828d7ffec9cd3ca4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:37.223858 1171 raft_consensus.cc:374] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:37.223919 1171 raft_consensus.cc:729] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 892780a68fd9443ebad27d673d6d0d23, State: Initialized, Role: FOLLOWER
I20240503 13:58:37.223946 1172 raft_consensus.cc:729] T 84c31335aee24600945210f193348807 P af52d7f74d294bc8828d7ffec9cd3ca4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af52d7f74d294bc8828d7ffec9cd3ca4, State: Initialized, Role: FOLLOWER
I20240503 13:58:37.224396 1171 consensus_queue.cc:260] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } }
I20240503 13:58:37.224396 1172 consensus_queue.cc:260] T 84c31335aee24600945210f193348807 P af52d7f74d294bc8828d7ffec9cd3ca4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } }
I20240503 13:58:37.225056 629 heartbeater.cc:503] Master 127.0.99.190:42995 was elected leader, sending a full tablet report...
I20240503 13:58:37.225275 1172 ts_tablet_manager.cc:1398] T 84c31335aee24600945210f193348807 P af52d7f74d294bc8828d7ffec9cd3ca4: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20240503 13:58:37.225296 1171 ts_tablet_manager.cc:1398] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 13:58:37.225808 1153 heartbeater.cc:503] Master 127.0.99.190:42995 was elected leader, sending a full tablet report...
I20240503 13:58:37.226902 1173 raft_consensus.cc:348] T 84c31335aee24600945210f193348807 P e16f7d9da90c4521a8e763bca0084376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } }
I20240503 13:58:37.227110 1173 raft_consensus.cc:374] T 84c31335aee24600945210f193348807 P e16f7d9da90c4521a8e763bca0084376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:37.227208 1173 raft_consensus.cc:729] T 84c31335aee24600945210f193348807 P e16f7d9da90c4521a8e763bca0084376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e16f7d9da90c4521a8e763bca0084376, State: Initialized, Role: FOLLOWER
I20240503 13:58:37.227710 1178 raft_consensus.cc:482] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:37.227684 1173 consensus_queue.cc:260] T 84c31335aee24600945210f193348807 P e16f7d9da90c4521a8e763bca0084376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } }
I20240503 13:58:37.227856 1178 raft_consensus.cc:504] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } }
I20240503 13:58:37.228483 1173 ts_tablet_manager.cc:1398] T 84c31335aee24600945210f193348807 P e16f7d9da90c4521a8e763bca0084376: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240503 13:58:37.228881 891 heartbeater.cc:503] Master 127.0.99.190:42995 was elected leader, sending a full tablet report...
I20240503 13:58:37.231210 1178 leader_election.cc:288] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e16f7d9da90c4521a8e763bca0084376 (127.0.99.131:44007), af52d7f74d294bc8828d7ffec9cd3ca4 (127.0.99.129:39031)
I20240503 13:58:37.233990 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84c31335aee24600945210f193348807" candidate_uuid: "892780a68fd9443ebad27d673d6d0d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16f7d9da90c4521a8e763bca0084376" is_pre_election: true
I20240503 13:58:37.234248 845 raft_consensus.cc:2454] T 84c31335aee24600945210f193348807 P e16f7d9da90c4521a8e763bca0084376 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 892780a68fd9443ebad27d673d6d0d23 in term 0.
I20240503 13:58:37.234706 1042 leader_election.cc:302] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 892780a68fd9443ebad27d673d6d0d23, e16f7d9da90c4521a8e763bca0084376; no voters:
I20240503 13:58:37.234982 1178 raft_consensus.cc:2781] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:37.235062 1178 raft_consensus.cc:482] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:37.235105 1178 raft_consensus.cc:3037] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:37.235185 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84c31335aee24600945210f193348807" candidate_uuid: "892780a68fd9443ebad27d673d6d0d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" is_pre_election: true
I20240503 13:58:37.235472 583 raft_consensus.cc:2454] T 84c31335aee24600945210f193348807 P af52d7f74d294bc8828d7ffec9cd3ca4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 892780a68fd9443ebad27d673d6d0d23 in term 0.
W20240503 13:58:37.259153 892 tablet.cc:2369] T 84c31335aee24600945210f193348807 P e16f7d9da90c4521a8e763bca0084376: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:58:37.270103 1178 raft_consensus.cc:504] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } }
I20240503 13:58:37.270737 1178 leader_election.cc:288] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [CANDIDATE]: Term 1 election: Requested vote from peers e16f7d9da90c4521a8e763bca0084376 (127.0.99.131:44007), af52d7f74d294bc8828d7ffec9cd3ca4 (127.0.99.129:39031)
I20240503 13:58:37.271227 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84c31335aee24600945210f193348807" candidate_uuid: "892780a68fd9443ebad27d673d6d0d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16f7d9da90c4521a8e763bca0084376"
I20240503 13:58:37.271410 845 raft_consensus.cc:3037] T 84c31335aee24600945210f193348807 P e16f7d9da90c4521a8e763bca0084376 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:37.271420 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84c31335aee24600945210f193348807" candidate_uuid: "892780a68fd9443ebad27d673d6d0d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4"
I20240503 13:58:37.271587 583 raft_consensus.cc:3037] T 84c31335aee24600945210f193348807 P af52d7f74d294bc8828d7ffec9cd3ca4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:37.272578 583 raft_consensus.cc:2454] T 84c31335aee24600945210f193348807 P af52d7f74d294bc8828d7ffec9cd3ca4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 892780a68fd9443ebad27d673d6d0d23 in term 1.
I20240503 13:58:37.272840 845 raft_consensus.cc:2454] T 84c31335aee24600945210f193348807 P e16f7d9da90c4521a8e763bca0084376 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 892780a68fd9443ebad27d673d6d0d23 in term 1.
I20240503 13:58:37.272958 1042 leader_election.cc:302] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 892780a68fd9443ebad27d673d6d0d23, af52d7f74d294bc8828d7ffec9cd3ca4; no voters:
I20240503 13:58:37.273175 1178 raft_consensus.cc:2781] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:37.273475 1178 raft_consensus.cc:686] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [term 1 LEADER]: Becoming Leader. State: Replica: 892780a68fd9443ebad27d673d6d0d23, State: Running, Role: LEADER
I20240503 13:58:37.273905 1178 consensus_queue.cc:237] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } }
I20240503 13:58:37.277632 440 catalog_manager.cc:5484] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 892780a68fd9443ebad27d673d6d0d23 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "892780a68fd9443ebad27d673d6d0d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:37.302403 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 13:58:37.304152 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver af52d7f74d294bc8828d7ffec9cd3ca4 to finish bootstrapping
I20240503 13:58:37.310084 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c72fe65a1c474fa1b0670828e00f6d27 to finish bootstrapping
I20240503 13:58:37.316196 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e16f7d9da90c4521a8e763bca0084376 to finish bootstrapping
I20240503 13:58:37.321215 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 45bbe6ca5c69471cbfaa002a655bdcd9 to finish bootstrapping
I20240503 13:58:37.326946 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 892780a68fd9443ebad27d673d6d0d23 to finish bootstrapping
W20240503 13:58:37.361652 630 tablet.cc:2369] T 84c31335aee24600945210f193348807 P af52d7f74d294bc8828d7ffec9cd3ca4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 13:58:37.409060 1154 tablet.cc:2369] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:58:37.528175 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:37.528649 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:37.530990 1085 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:58:37.532872 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:37.542912 823 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:37.564285 760 heartbeater.cc:503] Master 127.0.99.190:42995 was elected leader, sending a full tablet report...
I20240503 13:58:37.776876 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:37.777387 1085 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:58:37.779150 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:37.779778 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:37.792685 823 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:37.844959 1248 consensus_queue.cc:1035] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:37.850724 1251 consensus_queue.cc:1035] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:37.984261 1022 heartbeater.cc:503] Master 127.0.99.190:42995 was elected leader, sending a full tablet report...
I20240503 13:58:38.083690 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:38.088892 823 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:38.090127 1085 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:58:38.091921 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:38.101544 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:38.333949 1085 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:58:38.334689 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:38.336301 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:38.336968 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:38.339838 823 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:38.427253 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:38.428639 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:38.429143 1085 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:58:38.430409 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:38.439863 823 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:38.549432 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:38.550107 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:38.552397 1085 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:58:38.556403 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:38.564990 823 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:38.740963 440 ts_manager.cc:292] Set tserver state for af52d7f74d294bc8828d7ffec9cd3ca4 to MAINTENANCE_MODE
I20240503 13:58:38.932525 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:38.932974 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:38.933562 1085 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:58:38.935962 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:38.946106 823 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:39.025668 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:39.025820 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:39.031348 1085 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:58:39.032078 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:39.039786 823 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:39.140638 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:39.140924 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:39.142855 823 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:39.143747 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:39.143812 1085 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:58:39.188546 1107 consensus_queue.cc:237] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } attrs { replace: true } }
I20240503 13:58:39.189882 583 raft_consensus.cc:1261] T 84c31335aee24600945210f193348807 P af52d7f74d294bc8828d7ffec9cd3ca4 [term 1 FOLLOWER]: Refusing update from remote peer 892780a68fd9443ebad27d673d6d0d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:39.189910 845 raft_consensus.cc:1261] T 84c31335aee24600945210f193348807 P e16f7d9da90c4521a8e763bca0084376 [term 1 FOLLOWER]: Refusing update from remote peer 892780a68fd9443ebad27d673d6d0d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:39.190279 1504 consensus_queue.cc:1035] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:58:39.190459 1248 consensus_queue.cc:1035] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:58:39.192375 1504 raft_consensus.cc:2932] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } attrs { replace: true } } }
I20240503 13:58:39.192946 845 raft_consensus.cc:2932] T 84c31335aee24600945210f193348807 P e16f7d9da90c4521a8e763bca0084376 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } attrs { replace: true } } }
I20240503 13:58:39.193655 583 raft_consensus.cc:2932] T 84c31335aee24600945210f193348807 P af52d7f74d294bc8828d7ffec9cd3ca4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } attrs { replace: true } } }
I20240503 13:58:39.199345 440 catalog_manager.cc:5484] T 84c31335aee24600945210f193348807 P e16f7d9da90c4521a8e763bca0084376 reported cstate change: config changed from index -1 to 2. New cstate: current_term: 1 leader_uuid: "892780a68fd9443ebad27d673d6d0d23" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } attrs { replace: true } } }
I20240503 13:58:39.203092 1107 consensus_queue.cc:237] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } attrs { replace: true } } peers { permanent_uuid: "45bbe6ca5c69471cbfaa002a655bdcd9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 38173 } attrs { promote: true } }
I20240503 13:58:39.205013 583 raft_consensus.cc:1261] T 84c31335aee24600945210f193348807 P af52d7f74d294bc8828d7ffec9cd3ca4 [term 1 FOLLOWER]: Refusing update from remote peer 892780a68fd9443ebad27d673d6d0d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:58:39.205132 845 raft_consensus.cc:1261] T 84c31335aee24600945210f193348807 P e16f7d9da90c4521a8e763bca0084376 [term 1 FOLLOWER]: Refusing update from remote peer 892780a68fd9443ebad27d673d6d0d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:58:39.205503 1514 consensus_queue.cc:1035] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:58:39.205701 1248 consensus_queue.cc:1035] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:58:39.207302 1504 raft_consensus.cc:2932] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 45bbe6ca5c69471cbfaa002a655bdcd9 (127.0.99.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } attrs { replace: true } } peers { permanent_uuid: "45bbe6ca5c69471cbfaa002a655bdcd9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 38173 } attrs { promote: true } } }
I20240503 13:58:39.207695 845 raft_consensus.cc:2932] T 84c31335aee24600945210f193348807 P e16f7d9da90c4521a8e763bca0084376 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 45bbe6ca5c69471cbfaa002a655bdcd9 (127.0.99.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } attrs { replace: true } } peers { permanent_uuid: "45bbe6ca5c69471cbfaa002a655bdcd9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 38173 } attrs { promote: true } } }
I20240503 13:58:39.207791 583 raft_consensus.cc:2932] T 84c31335aee24600945210f193348807 P af52d7f74d294bc8828d7ffec9cd3ca4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 45bbe6ca5c69471cbfaa002a655bdcd9 (127.0.99.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } attrs { replace: true } } peers { permanent_uuid: "45bbe6ca5c69471cbfaa002a655bdcd9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 38173 } attrs { promote: true } } }
W20240503 13:58:39.209712 1042 consensus_peers.cc:485] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 -> Peer 45bbe6ca5c69471cbfaa002a655bdcd9 (127.0.99.132:38173): Couldn't send request to peer 45bbe6ca5c69471cbfaa002a655bdcd9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84c31335aee24600945210f193348807. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:39.284132 425 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 84c31335aee24600945210f193348807 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 13:58:39.284582 439 catalog_manager.cc:5484] T 84c31335aee24600945210f193348807 P e16f7d9da90c4521a8e763bca0084376 reported cstate change: config changed from index 2 to 3, NON_VOTER 45bbe6ca5c69471cbfaa002a655bdcd9 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "892780a68fd9443ebad27d673d6d0d23" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } attrs { replace: true } } peers { permanent_uuid: "45bbe6ca5c69471cbfaa002a655bdcd9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 38173 } attrs { promote: true } } }
I20240503 13:58:39.655629 1526 ts_tablet_manager.cc:902] T 84c31335aee24600945210f193348807 P 45bbe6ca5c69471cbfaa002a655bdcd9: Initiating tablet copy from peer 892780a68fd9443ebad27d673d6d0d23 (127.0.99.133:36003)
I20240503 13:58:39.656352 1526 tablet_copy_client.cc:308] T 84c31335aee24600945210f193348807 P 45bbe6ca5c69471cbfaa002a655bdcd9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:36003
I20240503 13:58:39.660212 1127 tablet_copy_service.cc:140] P 892780a68fd9443ebad27d673d6d0d23: Received BeginTabletCopySession request for tablet 84c31335aee24600945210f193348807 from peer 45bbe6ca5c69471cbfaa002a655bdcd9 ({username='slave'} at 127.0.99.132:57387)
I20240503 13:58:39.660346 1127 tablet_copy_service.cc:161] P 892780a68fd9443ebad27d673d6d0d23: Beginning new tablet copy session on tablet 84c31335aee24600945210f193348807 from peer 45bbe6ca5c69471cbfaa002a655bdcd9 at {username='slave'} at 127.0.99.132:57387: session id = 45bbe6ca5c69471cbfaa002a655bdcd9-84c31335aee24600945210f193348807
I20240503 13:58:39.661972 1127 tablet_copy_source_session.cc:205] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:58:39.665014 1526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84c31335aee24600945210f193348807. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:39.670383 1526 tablet_copy_client.cc:783] T 84c31335aee24600945210f193348807 P 45bbe6ca5c69471cbfaa002a655bdcd9: tablet copy: Starting download of 0 data blocks...
I20240503 13:58:39.670576 1526 tablet_copy_client.cc:647] T 84c31335aee24600945210f193348807 P 45bbe6ca5c69471cbfaa002a655bdcd9: tablet copy: Starting download of 1 WAL segments...
I20240503 13:58:39.672313 1526 tablet_copy_client.cc:523] T 84c31335aee24600945210f193348807 P 45bbe6ca5c69471cbfaa002a655bdcd9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:58:39.681066 1526 tablet_bootstrap.cc:492] T 84c31335aee24600945210f193348807 P 45bbe6ca5c69471cbfaa002a655bdcd9: Bootstrap starting.
I20240503 13:58:39.686539 1526 log.cc:824] T 84c31335aee24600945210f193348807 P 45bbe6ca5c69471cbfaa002a655bdcd9: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:39.689047 1526 tablet_bootstrap.cc:492] T 84c31335aee24600945210f193348807 P 45bbe6ca5c69471cbfaa002a655bdcd9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:58:39.689283 1526 tablet_bootstrap.cc:492] T 84c31335aee24600945210f193348807 P 45bbe6ca5c69471cbfaa002a655bdcd9: Bootstrap complete.
I20240503 13:58:39.689525 1526 ts_tablet_manager.cc:1367] T 84c31335aee24600945210f193348807 P 45bbe6ca5c69471cbfaa002a655bdcd9: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240503 13:58:39.692072 1526 raft_consensus.cc:348] T 84c31335aee24600945210f193348807 P 45bbe6ca5c69471cbfaa002a655bdcd9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } attrs { replace: true } } peers { permanent_uuid: "45bbe6ca5c69471cbfaa002a655bdcd9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 38173 } attrs { promote: true } }
I20240503 13:58:39.692353 1526 raft_consensus.cc:729] T 84c31335aee24600945210f193348807 P 45bbe6ca5c69471cbfaa002a655bdcd9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 45bbe6ca5c69471cbfaa002a655bdcd9, State: Initialized, Role: LEARNER
I20240503 13:58:39.692754 1526 consensus_queue.cc:260] T 84c31335aee24600945210f193348807 P 45bbe6ca5c69471cbfaa002a655bdcd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } attrs { replace: true } } peers { permanent_uuid: "45bbe6ca5c69471cbfaa002a655bdcd9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 38173 } attrs { promote: true } }
I20240503 13:58:39.693624 1526 ts_tablet_manager.cc:1398] T 84c31335aee24600945210f193348807 P 45bbe6ca5c69471cbfaa002a655bdcd9: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20240503 13:58:39.694386 1127 tablet_copy_service.cc:342] P 892780a68fd9443ebad27d673d6d0d23: Request end of tablet copy session 45bbe6ca5c69471cbfaa002a655bdcd9-84c31335aee24600945210f193348807 received from {username='slave'} at 127.0.99.132:57387
I20240503 13:58:39.694499 1127 tablet_copy_service.cc:433] P 892780a68fd9443ebad27d673d6d0d23: ending tablet copy session 45bbe6ca5c69471cbfaa002a655bdcd9-84c31335aee24600945210f193348807 on tablet 84c31335aee24600945210f193348807 with peer 45bbe6ca5c69471cbfaa002a655bdcd9
I20240503 13:58:40.142246 976 raft_consensus.cc:1203] T 84c31335aee24600945210f193348807 P 45bbe6ca5c69471cbfaa002a655bdcd9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20240503 13:58:40.738233 1561 raft_consensus.cc:1050] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23: attempting to promote NON_VOTER 45bbe6ca5c69471cbfaa002a655bdcd9 to VOTER
I20240503 13:58:40.738641 1561 consensus_queue.cc:237] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } attrs { replace: true } } peers { permanent_uuid: "45bbe6ca5c69471cbfaa002a655bdcd9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38173 } attrs { promote: false } }
I20240503 13:58:40.740262 583 raft_consensus.cc:1261] T 84c31335aee24600945210f193348807 P af52d7f74d294bc8828d7ffec9cd3ca4 [term 1 FOLLOWER]: Refusing update from remote peer 892780a68fd9443ebad27d673d6d0d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:58:40.740381 845 raft_consensus.cc:1261] T 84c31335aee24600945210f193348807 P e16f7d9da90c4521a8e763bca0084376 [term 1 FOLLOWER]: Refusing update from remote peer 892780a68fd9443ebad27d673d6d0d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:58:40.740368 976 raft_consensus.cc:1261] T 84c31335aee24600945210f193348807 P 45bbe6ca5c69471cbfaa002a655bdcd9 [term 1 LEARNER]: Refusing update from remote peer 892780a68fd9443ebad27d673d6d0d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:58:40.740710 1563 consensus_queue.cc:1035] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:58:40.741088 1564 consensus_queue.cc:1035] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45bbe6ca5c69471cbfaa002a655bdcd9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:58:40.741262 1545 consensus_queue.cc:1035] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:58:40.743793 1564 raft_consensus.cc:2932] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 45bbe6ca5c69471cbfaa002a655bdcd9 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } attrs { replace: true } } peers { permanent_uuid: "45bbe6ca5c69471cbfaa002a655bdcd9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38173 } attrs { promote: false } } }
I20240503 13:58:40.745513 845 raft_consensus.cc:2932] T 84c31335aee24600945210f193348807 P e16f7d9da90c4521a8e763bca0084376 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 45bbe6ca5c69471cbfaa002a655bdcd9 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } attrs { replace: true } } peers { permanent_uuid: "45bbe6ca5c69471cbfaa002a655bdcd9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38173 } attrs { promote: false } } }
I20240503 13:58:40.745589 583 raft_consensus.cc:2932] T 84c31335aee24600945210f193348807 P af52d7f74d294bc8828d7ffec9cd3ca4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 45bbe6ca5c69471cbfaa002a655bdcd9 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } attrs { replace: true } } peers { permanent_uuid: "45bbe6ca5c69471cbfaa002a655bdcd9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38173 } attrs { promote: false } } }
I20240503 13:58:40.745834 976 raft_consensus.cc:2932] T 84c31335aee24600945210f193348807 P 45bbe6ca5c69471cbfaa002a655bdcd9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 45bbe6ca5c69471cbfaa002a655bdcd9 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } attrs { replace: true } } peers { permanent_uuid: "45bbe6ca5c69471cbfaa002a655bdcd9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38173 } attrs { promote: false } } }
I20240503 13:58:40.747799 441 catalog_manager.cc:5484] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 reported cstate change: config changed from index 3 to 4, 45bbe6ca5c69471cbfaa002a655bdcd9 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "892780a68fd9443ebad27d673d6d0d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af52d7f74d294bc8828d7ffec9cd3ca4" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39031 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45bbe6ca5c69471cbfaa002a655bdcd9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38173 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:40.752408 1107 consensus_queue.cc:237] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "45bbe6ca5c69471cbfaa002a655bdcd9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38173 } attrs { promote: false } }
I20240503 13:58:40.753454 845 raft_consensus.cc:1261] T 84c31335aee24600945210f193348807 P e16f7d9da90c4521a8e763bca0084376 [term 1 FOLLOWER]: Refusing update from remote peer 892780a68fd9443ebad27d673d6d0d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:58:40.753563 976 raft_consensus.cc:1261] T 84c31335aee24600945210f193348807 P 45bbe6ca5c69471cbfaa002a655bdcd9 [term 1 FOLLOWER]: Refusing update from remote peer 892780a68fd9443ebad27d673d6d0d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:58:40.753813 1564 consensus_queue.cc:1035] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:58:40.753952 1563 consensus_queue.cc:1035] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45bbe6ca5c69471cbfaa002a655bdcd9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:58:40.755548 1561 raft_consensus.cc:2932] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER af52d7f74d294bc8828d7ffec9cd3ca4 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "45bbe6ca5c69471cbfaa002a655bdcd9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38173 } attrs { promote: false } } }
I20240503 13:58:40.755986 845 raft_consensus.cc:2932] T 84c31335aee24600945210f193348807 P e16f7d9da90c4521a8e763bca0084376 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER af52d7f74d294bc8828d7ffec9cd3ca4 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "45bbe6ca5c69471cbfaa002a655bdcd9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38173 } attrs { promote: false } } }
I20240503 13:58:40.756039 976 raft_consensus.cc:2932] T 84c31335aee24600945210f193348807 P 45bbe6ca5c69471cbfaa002a655bdcd9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER af52d7f74d294bc8828d7ffec9cd3ca4 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } } peers { permanent_uuid: "45bbe6ca5c69471cbfaa002a655bdcd9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38173 } attrs { promote: false } } }
I20240503 13:58:40.757573 425 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 84c31335aee24600945210f193348807 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:58:40.758122 437 catalog_manager.cc:5484] T 84c31335aee24600945210f193348807 P 892780a68fd9443ebad27d673d6d0d23 reported cstate change: config changed from index 4 to 5, VOTER af52d7f74d294bc8828d7ffec9cd3ca4 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "892780a68fd9443ebad27d673d6d0d23" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e16f7d9da90c4521a8e763bca0084376" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "892780a68fd9443ebad27d673d6d0d23" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45bbe6ca5c69471cbfaa002a655bdcd9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38173 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:40.764612 563 tablet_service.cc:1514] Processing DeleteTablet for tablet 84c31335aee24600945210f193348807 with delete_type TABLET_DATA_TOMBSTONED (TS af52d7f74d294bc8828d7ffec9cd3ca4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54404
I20240503 13:58:40.765120 1575 tablet_replica.cc:330] T 84c31335aee24600945210f193348807 P af52d7f74d294bc8828d7ffec9cd3ca4: stopping tablet replica
I20240503 13:58:40.765436 1575 raft_consensus.cc:2229] T 84c31335aee24600945210f193348807 P af52d7f74d294bc8828d7ffec9cd3ca4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:58:40.765720 1575 raft_consensus.cc:2258] T 84c31335aee24600945210f193348807 P af52d7f74d294bc8828d7ffec9cd3ca4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:58:40.766724 1575 ts_tablet_manager.cc:1875] T 84c31335aee24600945210f193348807 P af52d7f74d294bc8828d7ffec9cd3ca4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:58:40.768604 1575 ts_tablet_manager.cc:1888] T 84c31335aee24600945210f193348807 P af52d7f74d294bc8828d7ffec9cd3ca4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 13:58:40.768684 1575 log.cc:1196] T 84c31335aee24600945210f193348807 P af52d7f74d294bc8828d7ffec9cd3ca4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/84c31335aee24600945210f193348807
I20240503 13:58:40.769271 427 catalog_manager.cc:4830] TS af52d7f74d294bc8828d7ffec9cd3ca4 (127.0.99.129:39031): tablet 84c31335aee24600945210f193348807 (table TestTable [id=13101edcc3bf4c30b227146df0866b3e]) successfully deleted
I20240503 13:58:41.101409 1085 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:58:41.102154 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:41.103499 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:41.105607 823 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:41.112120 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:41.183347 1085 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:58:41.183562 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:41.184883 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:41.191923 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:41.193337 823 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:41.260241 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:41.261217 1085 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:58:41.261759 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:41.269520 823 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:41.269796 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:41.300107 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 632
I20240503 13:58:41.480227 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:41.480324 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:41.480540 1085 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:58:41.482654 823 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:41.552388 1085 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:58:41.553587 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:41.553882 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:41.554068 823 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:41.646668 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:41.649145 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:41.649297 823 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:41.656078 1085 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:58:41.722215 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:41.722694 823 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:41.723902 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:41.724128 1085 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:58:41.755825 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 501
I20240503 13:58:41.763171 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 763
I20240503 13:58:41.769317 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 894
W20240503 13:58:41.774787 1042 connection.cc:535] client connection to 127.0.99.132:38173 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 13:58:41.774946 1042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 13:58:41.775215 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 1025
I20240503 13:58:41.780987 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 410
2024-05-03T13:58:41Z chronyd exiting
[ OK ] RebalanceIgnoredTserversTest.Basic (5925 ms)
[----------] 1 test from RebalanceIgnoredTserversTest (5926 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/1
I20240503 13:58:41.803562 398 test_util.cc:268] Using random seed: 503712037
I20240503 13:58:41.804291 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 13:58:41.804337 398 ts_itest-base.cc:113] --------------
I20240503 13:58:41.804348 398 ts_itest-base.cc:114] 5 tablet servers
I20240503 13:58:41.804359 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 13:58:41.804383 398 ts_itest-base.cc:116] --------------
2024-05-03T13:58:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T13:58:41Z Disabled control of system clock
I20240503 13:58:41.811931 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:33621
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:33621
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 13:58:41.910918 1796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:41.911157 1796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:41.911229 1796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:41.914115 1796 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:41.914180 1796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 13:58:41.914216 1796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:41.914280 1796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 13:58:41.914321 1796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 13:58:41.917779 1796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:33621
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:33621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.1796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:41.918668 1796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:41.919814 1796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:41.927459 1802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:41.928170 1801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:41.928897 1804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:41.929708 1796 server_base.cc:1034] running on GCE node
I20240503 13:58:41.930184 1796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:41.931059 1796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:41.932204 1796 hybrid_clock.cc:648] HybridClock initialized: now 1714744721932177 us; error 44 us; skew 500 ppm
I20240503 13:58:41.933882 1796 webserver.cc:460] Webserver started at http://127.0.99.190:36265/ using document root <none> and password file <none>
I20240503 13:58:41.934388 1796 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:41.934446 1796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:41.934602 1796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:41.936115 1796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "142ddd81de1f4946b83aa51d388426dd"
format_stamp: "Formatted at 2024-05-03 13:58:41 on dist-test-slave-fr9h"
I20240503 13:58:41.936627 1796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "142ddd81de1f4946b83aa51d388426dd"
format_stamp: "Formatted at 2024-05-03 13:58:41 on dist-test-slave-fr9h"
I20240503 13:58:41.939630 1796 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 13:58:41.941649 1810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:41.942523 1796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:41.942633 1796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "142ddd81de1f4946b83aa51d388426dd"
format_stamp: "Formatted at 2024-05-03 13:58:41 on dist-test-slave-fr9h"
I20240503 13:58:41.942709 1796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:41.972651 1796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:41.973224 1796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:41.973372 1796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:41.978885 1796 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:33621
I20240503 13:58:41.978897 1862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:33621 every 8 connection(s)
I20240503 13:58:41.979779 1796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 13:58:41.982113 1863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:41.986775 1863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 142ddd81de1f4946b83aa51d388426dd: Bootstrap starting.
I20240503 13:58:41.987123 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 1796
I20240503 13:58:41.987248 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 13:58:41.989080 1863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 142ddd81de1f4946b83aa51d388426dd: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:41.989702 1863 log.cc:824] T 00000000000000000000000000000000 P 142ddd81de1f4946b83aa51d388426dd: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:41.991423 1863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 142ddd81de1f4946b83aa51d388426dd: No bootstrap required, opened a new log
I20240503 13:58:41.994244 1863 raft_consensus.cc:348] T 00000000000000000000000000000000 P 142ddd81de1f4946b83aa51d388426dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142ddd81de1f4946b83aa51d388426dd" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 33621 } }
I20240503 13:58:41.994437 1863 raft_consensus.cc:374] T 00000000000000000000000000000000 P 142ddd81de1f4946b83aa51d388426dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:41.994500 1863 raft_consensus.cc:729] T 00000000000000000000000000000000 P 142ddd81de1f4946b83aa51d388426dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142ddd81de1f4946b83aa51d388426dd, State: Initialized, Role: FOLLOWER
I20240503 13:58:41.994910 1863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 142ddd81de1f4946b83aa51d388426dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142ddd81de1f4946b83aa51d388426dd" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 33621 } }
I20240503 13:58:41.995038 1863 raft_consensus.cc:388] T 00000000000000000000000000000000 P 142ddd81de1f4946b83aa51d388426dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:58:41.995102 1863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 142ddd81de1f4946b83aa51d388426dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:58:41.995216 1863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 142ddd81de1f4946b83aa51d388426dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:41.996071 1863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 142ddd81de1f4946b83aa51d388426dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142ddd81de1f4946b83aa51d388426dd" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 33621 } }
I20240503 13:58:41.996382 1863 leader_election.cc:302] T 00000000000000000000000000000000 P 142ddd81de1f4946b83aa51d388426dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 142ddd81de1f4946b83aa51d388426dd; no voters:
I20240503 13:58:41.996598 1863 leader_election.cc:288] T 00000000000000000000000000000000 P 142ddd81de1f4946b83aa51d388426dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:58:41.996692 1868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 142ddd81de1f4946b83aa51d388426dd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:41.996906 1868 raft_consensus.cc:686] T 00000000000000000000000000000000 P 142ddd81de1f4946b83aa51d388426dd [term 1 LEADER]: Becoming Leader. State: Replica: 142ddd81de1f4946b83aa51d388426dd, State: Running, Role: LEADER
I20240503 13:58:41.997166 1868 consensus_queue.cc:237] T 00000000000000000000000000000000 P 142ddd81de1f4946b83aa51d388426dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142ddd81de1f4946b83aa51d388426dd" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 33621 } }
I20240503 13:58:41.997552 1863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 142ddd81de1f4946b83aa51d388426dd [sys.catalog]: configured and running, proceeding with master startup.
I20240503 13:58:41.998589 1869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 142ddd81de1f4946b83aa51d388426dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "142ddd81de1f4946b83aa51d388426dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142ddd81de1f4946b83aa51d388426dd" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 33621 } } }
I20240503 13:58:41.998620 1870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 142ddd81de1f4946b83aa51d388426dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 142ddd81de1f4946b83aa51d388426dd. Latest consensus state: current_term: 1 leader_uuid: "142ddd81de1f4946b83aa51d388426dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142ddd81de1f4946b83aa51d388426dd" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 33621 } } }
I20240503 13:58:41.998802 1870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 142ddd81de1f4946b83aa51d388426dd [sys.catalog]: This master's current role is: LEADER
I20240503 13:58:41.999043 1869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 142ddd81de1f4946b83aa51d388426dd [sys.catalog]: This master's current role is: LEADER
I20240503 13:58:41.999183 1877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 13:58:42.002007 1877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 13:58:42.006994 1877 catalog_manager.cc:1347] Generated new cluster ID: bf22ca89ca254f7cabff73d369318208
I20240503 13:58:42.007057 1877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 13:58:42.026912 1877 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 13:58:42.027817 1877 catalog_manager.cc:1504] Loading token signing keys...
I20240503 13:58:42.036278 1877 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 142ddd81de1f4946b83aa51d388426dd: Generated new TSK 0
I20240503 13:58:42.036919 1877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 13:58:42.045476 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33621
--builtin_ntp_servers=127.0.99.148:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 13:58:42.153380 1887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:42.153600 1887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:42.153672 1887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:42.156618 1887 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:42.156690 1887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:42.156783 1887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 13:58:42.160287 1887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33621
--never_fsync=true
--heap_profile_path=/tmp/kudu.1887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:42.161141 1887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:42.162251 1887 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:42.170816 1893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:42.171612 1892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:42.172389 1895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:42.173213 1887 server_base.cc:1034] running on GCE node
I20240503 13:58:42.173563 1887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:42.174064 1887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:42.175216 1887 hybrid_clock.cc:648] HybridClock initialized: now 1714744722175201 us; error 29 us; skew 500 ppm
I20240503 13:58:42.176970 1887 webserver.cc:460] Webserver started at http://127.0.99.129:37815/ using document root <none> and password file <none>
I20240503 13:58:42.177886 1887 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:42.177938 1887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:42.178138 1887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:42.179755 1887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "5dec3586cf4b4bf9801dee83df27167b"
format_stamp: "Formatted at 2024-05-03 13:58:42 on dist-test-slave-fr9h"
I20240503 13:58:42.180230 1887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "5dec3586cf4b4bf9801dee83df27167b"
format_stamp: "Formatted at 2024-05-03 13:58:42 on dist-test-slave-fr9h"
I20240503 13:58:42.183382 1887 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 13:58:42.185434 1901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:42.186380 1887 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:42.186489 1887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "5dec3586cf4b4bf9801dee83df27167b"
format_stamp: "Formatted at 2024-05-03 13:58:42 on dist-test-slave-fr9h"
I20240503 13:58:42.186570 1887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:42.230859 1887 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:42.231527 1887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:42.231689 1887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:42.232203 1887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:42.233068 1887 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:42.233117 1887 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:42.233175 1887 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:42.233207 1887 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:42.241242 1887 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:41733
I20240503 13:58:42.241281 2014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:41733 every 8 connection(s)
I20240503 13:58:42.242112 1887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 13:58:42.250823 2015 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33621
I20240503 13:58:42.251024 2015 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:42.251551 2015 heartbeater.cc:511] Master 127.0.99.190:33621 requested a full tablet report, sending...
I20240503 13:58:42.251895 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 1887
I20240503 13:58:42.251989 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 13:58:42.253013 1827 ts_manager.cc:194] Registered new tserver with Master: 5dec3586cf4b4bf9801dee83df27167b (127.0.99.129:41733)
I20240503 13:58:42.254211 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33621
--builtin_ntp_servers=127.0.99.148:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:58:42.254500 1827 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:41469
W20240503 13:58:42.357873 2018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:42.358064 2018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:42.358104 2018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:42.361080 2018 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:42.361143 2018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:42.361205 2018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 13:58:42.364532 2018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33621
--never_fsync=true
--heap_profile_path=/tmp/kudu.2018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:42.365317 2018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:42.366313 2018 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:42.374977 2024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:42.375630 2023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:42.376453 2026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:42.377207 2018 server_base.cc:1034] running on GCE node
I20240503 13:58:42.377656 2018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:42.378155 2018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:42.379326 2018 hybrid_clock.cc:648] HybridClock initialized: now 1714744722379297 us; error 40 us; skew 500 ppm
I20240503 13:58:42.381121 2018 webserver.cc:460] Webserver started at http://127.0.99.130:40311/ using document root <none> and password file <none>
I20240503 13:58:42.381678 2018 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:42.381733 2018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:42.381918 2018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:42.383577 2018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f485fd5312ae46348cd6a664484b4510"
format_stamp: "Formatted at 2024-05-03 13:58:42 on dist-test-slave-fr9h"
I20240503 13:58:42.384037 2018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f485fd5312ae46348cd6a664484b4510"
format_stamp: "Formatted at 2024-05-03 13:58:42 on dist-test-slave-fr9h"
I20240503 13:58:42.387081 2018 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 13:58:42.389104 2032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:42.390120 2018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:42.390218 2018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f485fd5312ae46348cd6a664484b4510"
format_stamp: "Formatted at 2024-05-03 13:58:42 on dist-test-slave-fr9h"
I20240503 13:58:42.390287 2018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:42.419169 2018 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:42.419734 2018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:42.419912 2018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:42.420420 2018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:42.421238 2018 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:42.421284 2018 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:42.421325 2018 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:42.421370 2018 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:42.429227 2018 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:36393
I20240503 13:58:42.429265 2145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:36393 every 8 connection(s)
I20240503 13:58:42.430107 2018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 13:58:42.437805 2146 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33621
I20240503 13:58:42.438002 2146 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:42.438458 2146 heartbeater.cc:511] Master 127.0.99.190:33621 requested a full tablet report, sending...
I20240503 13:58:42.439350 1827 ts_manager.cc:194] Registered new tserver with Master: f485fd5312ae46348cd6a664484b4510 (127.0.99.130:36393)
I20240503 13:58:42.439780 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 2018
I20240503 13:58:42.439869 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 13:58:42.439878 1827 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:52141
I20240503 13:58:42.441753 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33621
--builtin_ntp_servers=127.0.99.148:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 13:58:42.542512 2149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:42.542701 2149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:42.542742 2149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:42.545506 2149 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:42.545559 2149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:42.545640 2149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 13:58:42.549029 2149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33621
--never_fsync=true
--heap_profile_path=/tmp/kudu.2149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:42.549809 2149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:42.550879 2149 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:42.559324 2157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:42.560165 2154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:42.560974 2155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:42.561776 2149 server_base.cc:1034] running on GCE node
I20240503 13:58:42.562129 2149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:42.562629 2149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:42.563772 2149 hybrid_clock.cc:648] HybridClock initialized: now 1714744722563751 us; error 33 us; skew 500 ppm
I20240503 13:58:42.565537 2149 webserver.cc:460] Webserver started at http://127.0.99.131:34329/ using document root <none> and password file <none>
I20240503 13:58:42.566069 2149 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:42.566143 2149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:42.566337 2149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:42.568202 2149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ff84bd968a714ba8bd790788b7f906ae"
format_stamp: "Formatted at 2024-05-03 13:58:42 on dist-test-slave-fr9h"
I20240503 13:58:42.568727 2149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ff84bd968a714ba8bd790788b7f906ae"
format_stamp: "Formatted at 2024-05-03 13:58:42 on dist-test-slave-fr9h"
I20240503 13:58:42.572896 2149 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 13:58:42.575882 2163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:42.577081 2149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 13:58:42.577203 2149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ff84bd968a714ba8bd790788b7f906ae"
format_stamp: "Formatted at 2024-05-03 13:58:42 on dist-test-slave-fr9h"
I20240503 13:58:42.577297 2149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:42.599701 2149 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:42.600302 2149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:42.600462 2149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:42.600970 2149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:42.601810 2149 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:42.601859 2149 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:42.601933 2149 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:42.601959 2149 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:42.609889 2149 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:38155
I20240503 13:58:42.609961 2276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:38155 every 8 connection(s)
I20240503 13:58:42.610808 2149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 13:58:42.616909 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 2149
I20240503 13:58:42.617007 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 13:58:42.618983 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33621
--builtin_ntp_servers=127.0.99.148:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:58:42.619192 2277 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33621
I20240503 13:58:42.619412 2277 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:42.619890 2277 heartbeater.cc:511] Master 127.0.99.190:33621 requested a full tablet report, sending...
I20240503 13:58:42.620893 1827 ts_manager.cc:194] Registered new tserver with Master: ff84bd968a714ba8bd790788b7f906ae (127.0.99.131:38155)
I20240503 13:58:42.621630 1827 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:55077
W20240503 13:58:42.723166 2280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:42.723430 2280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:42.723508 2280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:42.726272 2280 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:42.726327 2280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:42.726384 2280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 13:58:42.729462 2280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33621
--never_fsync=true
--heap_profile_path=/tmp/kudu.2280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:42.730201 2280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:42.731186 2280 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:42.739185 2286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:42.739993 2285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:42.740726 2288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:42.741511 2280 server_base.cc:1034] running on GCE node
I20240503 13:58:42.741835 2280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:42.742344 2280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:42.743485 2280 hybrid_clock.cc:648] HybridClock initialized: now 1714744722743470 us; error 26 us; skew 500 ppm
I20240503 13:58:42.745204 2280 webserver.cc:460] Webserver started at http://127.0.99.132:38317/ using document root <none> and password file <none>
I20240503 13:58:42.745743 2280 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:42.745803 2280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:42.745985 2280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:42.747615 2280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "2d842902ff4d471581aef2afd45e74f5"
format_stamp: "Formatted at 2024-05-03 13:58:42 on dist-test-slave-fr9h"
I20240503 13:58:42.748091 2280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "2d842902ff4d471581aef2afd45e74f5"
format_stamp: "Formatted at 2024-05-03 13:58:42 on dist-test-slave-fr9h"
I20240503 13:58:42.751196 2280 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 13:58:42.753219 2294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:42.754104 2280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:42.754199 2280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2d842902ff4d471581aef2afd45e74f5"
format_stamp: "Formatted at 2024-05-03 13:58:42 on dist-test-slave-fr9h"
I20240503 13:58:42.754271 2280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:42.769109 2280 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:42.769688 2280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:42.769834 2280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:42.770340 2280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:42.771163 2280 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:42.771211 2280 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:42.771273 2280 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:42.771322 2280 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:42.779434 2280 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:40029
I20240503 13:58:42.779482 2407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:40029 every 8 connection(s)
I20240503 13:58:42.780378 2280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 13:58:42.784338 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 2280
I20240503 13:58:42.784448 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 13:58:42.786357 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33621
--builtin_ntp_servers=127.0.99.148:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:58:42.788661 2408 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33621
I20240503 13:58:42.788865 2408 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:42.789331 2408 heartbeater.cc:511] Master 127.0.99.190:33621 requested a full tablet report, sending...
I20240503 13:58:42.790479 1827 ts_manager.cc:194] Registered new tserver with Master: 2d842902ff4d471581aef2afd45e74f5 (127.0.99.132:40029)
I20240503 13:58:42.791203 1827 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:37127
W20240503 13:58:42.891378 2411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:42.891607 2411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:42.891669 2411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:42.894443 2411 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:42.894517 2411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:42.894599 2411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 13:58:42.898011 2411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33621
--never_fsync=true
--heap_profile_path=/tmp/kudu.2411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:42.898890 2411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:42.900031 2411 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:42.908282 2419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:42.909089 2416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:42.909880 2417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:42.910708 2411 server_base.cc:1034] running on GCE node
I20240503 13:58:42.911020 2411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:42.911564 2411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:42.912706 2411 hybrid_clock.cc:648] HybridClock initialized: now 1714744722912693 us; error 27 us; skew 500 ppm
I20240503 13:58:42.914489 2411 webserver.cc:460] Webserver started at http://127.0.99.133:42471/ using document root <none> and password file <none>
I20240503 13:58:42.914997 2411 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:42.915050 2411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:42.915220 2411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:42.916787 2411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "7823d31eecb64f8ba2b8094cb0db8132"
format_stamp: "Formatted at 2024-05-03 13:58:42 on dist-test-slave-fr9h"
I20240503 13:58:42.917236 2411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "7823d31eecb64f8ba2b8094cb0db8132"
format_stamp: "Formatted at 2024-05-03 13:58:42 on dist-test-slave-fr9h"
I20240503 13:58:42.920404 2411 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 13:58:42.922498 2425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:42.923480 2411 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:58:42.923583 2411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "7823d31eecb64f8ba2b8094cb0db8132"
format_stamp: "Formatted at 2024-05-03 13:58:42 on dist-test-slave-fr9h"
I20240503 13:58:42.923657 2411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:42.948987 2411 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:42.949520 2411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:42.949667 2411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:42.950165 2411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:42.950991 2411 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:42.951040 2411 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:42.951097 2411 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:42.951129 2411 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:42.959722 2411 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:35971
I20240503 13:58:42.959757 2538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:35971 every 8 connection(s)
I20240503 13:58:42.960636 2411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 13:58:42.961876 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 2411
I20240503 13:58:42.961987 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 13:58:42.968715 2539 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33621
I20240503 13:58:42.968922 2539 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:42.969432 2539 heartbeater.cc:511] Master 127.0.99.190:33621 requested a full tablet report, sending...
I20240503 13:58:42.970376 1827 ts_manager.cc:194] Registered new tserver with Master: 7823d31eecb64f8ba2b8094cb0db8132 (127.0.99.133:35971)
I20240503 13:58:42.970929 1827 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:41801
I20240503 13:58:42.975852 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 13:58:42.984608 1827 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36820:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 13:58:43.002352 2080 tablet_service.cc:1467] Processing CreateTablet for tablet c28b234ec8a5467eb43c194f41680db4 (DEFAULT_TABLE table=TestTable [id=fbd469b720af4420b7d0a3304656ba0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:43.002671 1949 tablet_service.cc:1467] Processing CreateTablet for tablet c28b234ec8a5467eb43c194f41680db4 (DEFAULT_TABLE table=TestTable [id=fbd469b720af4420b7d0a3304656ba0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:43.003428 2080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c28b234ec8a5467eb43c194f41680db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:43.003525 1949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c28b234ec8a5467eb43c194f41680db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:43.003259 2211 tablet_service.cc:1467] Processing CreateTablet for tablet c28b234ec8a5467eb43c194f41680db4 (DEFAULT_TABLE table=TestTable [id=fbd469b720af4420b7d0a3304656ba0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:43.004101 2211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c28b234ec8a5467eb43c194f41680db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:43.008772 2556 tablet_bootstrap.cc:492] T c28b234ec8a5467eb43c194f41680db4 P ff84bd968a714ba8bd790788b7f906ae: Bootstrap starting.
I20240503 13:58:43.009634 2558 tablet_bootstrap.cc:492] T c28b234ec8a5467eb43c194f41680db4 P f485fd5312ae46348cd6a664484b4510: Bootstrap starting.
I20240503 13:58:43.011106 2556 tablet_bootstrap.cc:654] T c28b234ec8a5467eb43c194f41680db4 P ff84bd968a714ba8bd790788b7f906ae: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:43.011456 2558 tablet_bootstrap.cc:654] T c28b234ec8a5467eb43c194f41680db4 P f485fd5312ae46348cd6a664484b4510: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:43.011601 2557 tablet_bootstrap.cc:492] T c28b234ec8a5467eb43c194f41680db4 P 5dec3586cf4b4bf9801dee83df27167b: Bootstrap starting.
I20240503 13:58:43.011829 2556 log.cc:824] T c28b234ec8a5467eb43c194f41680db4 P ff84bd968a714ba8bd790788b7f906ae: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:43.012166 2558 log.cc:824] T c28b234ec8a5467eb43c194f41680db4 P f485fd5312ae46348cd6a664484b4510: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:43.013315 2556 tablet_bootstrap.cc:492] T c28b234ec8a5467eb43c194f41680db4 P ff84bd968a714ba8bd790788b7f906ae: No bootstrap required, opened a new log
I20240503 13:58:43.013406 2557 tablet_bootstrap.cc:654] T c28b234ec8a5467eb43c194f41680db4 P 5dec3586cf4b4bf9801dee83df27167b: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:43.013484 2556 ts_tablet_manager.cc:1367] T c28b234ec8a5467eb43c194f41680db4 P ff84bd968a714ba8bd790788b7f906ae: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 13:58:43.013795 2558 tablet_bootstrap.cc:492] T c28b234ec8a5467eb43c194f41680db4 P f485fd5312ae46348cd6a664484b4510: No bootstrap required, opened a new log
I20240503 13:58:43.013953 2558 ts_tablet_manager.cc:1367] T c28b234ec8a5467eb43c194f41680db4 P f485fd5312ae46348cd6a664484b4510: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 13:58:43.014084 2557 log.cc:824] T c28b234ec8a5467eb43c194f41680db4 P 5dec3586cf4b4bf9801dee83df27167b: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:43.015645 2557 tablet_bootstrap.cc:492] T c28b234ec8a5467eb43c194f41680db4 P 5dec3586cf4b4bf9801dee83df27167b: No bootstrap required, opened a new log
I20240503 13:58:43.016116 2557 ts_tablet_manager.cc:1367] T c28b234ec8a5467eb43c194f41680db4 P 5dec3586cf4b4bf9801dee83df27167b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 13:58:43.016625 2556 raft_consensus.cc:348] T c28b234ec8a5467eb43c194f41680db4 P ff84bd968a714ba8bd790788b7f906ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dec3586cf4b4bf9801dee83df27167b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41733 } } peers { permanent_uuid: "f485fd5312ae46348cd6a664484b4510" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36393 } } peers { permanent_uuid: "ff84bd968a714ba8bd790788b7f906ae" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38155 } }
I20240503 13:58:43.016796 2556 raft_consensus.cc:374] T c28b234ec8a5467eb43c194f41680db4 P ff84bd968a714ba8bd790788b7f906ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:43.016855 2556 raft_consensus.cc:729] T c28b234ec8a5467eb43c194f41680db4 P ff84bd968a714ba8bd790788b7f906ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff84bd968a714ba8bd790788b7f906ae, State: Initialized, Role: FOLLOWER
I20240503 13:58:43.016925 2558 raft_consensus.cc:348] T c28b234ec8a5467eb43c194f41680db4 P f485fd5312ae46348cd6a664484b4510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dec3586cf4b4bf9801dee83df27167b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41733 } } peers { permanent_uuid: "f485fd5312ae46348cd6a664484b4510" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36393 } } peers { permanent_uuid: "ff84bd968a714ba8bd790788b7f906ae" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38155 } }
I20240503 13:58:43.017131 2558 raft_consensus.cc:374] T c28b234ec8a5467eb43c194f41680db4 P f485fd5312ae46348cd6a664484b4510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:43.017189 2558 raft_consensus.cc:729] T c28b234ec8a5467eb43c194f41680db4 P f485fd5312ae46348cd6a664484b4510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f485fd5312ae46348cd6a664484b4510, State: Initialized, Role: FOLLOWER
I20240503 13:58:43.017277 2556 consensus_queue.cc:260] T c28b234ec8a5467eb43c194f41680db4 P ff84bd968a714ba8bd790788b7f906ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dec3586cf4b4bf9801dee83df27167b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41733 } } peers { permanent_uuid: "f485fd5312ae46348cd6a664484b4510" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36393 } } peers { permanent_uuid: "ff84bd968a714ba8bd790788b7f906ae" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38155 } }
I20240503 13:58:43.017594 2558 consensus_queue.cc:260] T c28b234ec8a5467eb43c194f41680db4 P f485fd5312ae46348cd6a664484b4510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dec3586cf4b4bf9801dee83df27167b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41733 } } peers { permanent_uuid: "f485fd5312ae46348cd6a664484b4510" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36393 } } peers { permanent_uuid: "ff84bd968a714ba8bd790788b7f906ae" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38155 } }
I20240503 13:58:43.017916 2277 heartbeater.cc:503] Master 127.0.99.190:33621 was elected leader, sending a full tablet report...
I20240503 13:58:43.018064 2556 ts_tablet_manager.cc:1398] T c28b234ec8a5467eb43c194f41680db4 P ff84bd968a714ba8bd790788b7f906ae: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20240503 13:58:43.018374 2558 ts_tablet_manager.cc:1398] T c28b234ec8a5467eb43c194f41680db4 P f485fd5312ae46348cd6a664484b4510: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20240503 13:58:43.018697 2146 heartbeater.cc:503] Master 127.0.99.190:33621 was elected leader, sending a full tablet report...
I20240503 13:58:43.019416 2557 raft_consensus.cc:348] T c28b234ec8a5467eb43c194f41680db4 P 5dec3586cf4b4bf9801dee83df27167b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dec3586cf4b4bf9801dee83df27167b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41733 } } peers { permanent_uuid: "f485fd5312ae46348cd6a664484b4510" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36393 } } peers { permanent_uuid: "ff84bd968a714ba8bd790788b7f906ae" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38155 } }
I20240503 13:58:43.019578 2557 raft_consensus.cc:374] T c28b234ec8a5467eb43c194f41680db4 P 5dec3586cf4b4bf9801dee83df27167b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:43.019631 2557 raft_consensus.cc:729] T c28b234ec8a5467eb43c194f41680db4 P 5dec3586cf4b4bf9801dee83df27167b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dec3586cf4b4bf9801dee83df27167b, State: Initialized, Role: FOLLOWER
I20240503 13:58:43.020001 2557 consensus_queue.cc:260] T c28b234ec8a5467eb43c194f41680db4 P 5dec3586cf4b4bf9801dee83df27167b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dec3586cf4b4bf9801dee83df27167b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41733 } } peers { permanent_uuid: "f485fd5312ae46348cd6a664484b4510" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36393 } } peers { permanent_uuid: "ff84bd968a714ba8bd790788b7f906ae" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38155 } }
I20240503 13:58:43.020949 2015 heartbeater.cc:503] Master 127.0.99.190:33621 was elected leader, sending a full tablet report...
I20240503 13:58:43.021001 2557 ts_tablet_manager.cc:1398] T c28b234ec8a5467eb43c194f41680db4 P 5dec3586cf4b4bf9801dee83df27167b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 13:58:43.103101 2563 raft_consensus.cc:482] T c28b234ec8a5467eb43c194f41680db4 P f485fd5312ae46348cd6a664484b4510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:43.103298 2563 raft_consensus.cc:504] T c28b234ec8a5467eb43c194f41680db4 P f485fd5312ae46348cd6a664484b4510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dec3586cf4b4bf9801dee83df27167b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41733 } } peers { permanent_uuid: "f485fd5312ae46348cd6a664484b4510" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36393 } } peers { permanent_uuid: "ff84bd968a714ba8bd790788b7f906ae" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38155 } }
I20240503 13:58:43.104382 2563 leader_election.cc:288] T c28b234ec8a5467eb43c194f41680db4 P f485fd5312ae46348cd6a664484b4510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dec3586cf4b4bf9801dee83df27167b (127.0.99.129:41733), ff84bd968a714ba8bd790788b7f906ae (127.0.99.131:38155)
I20240503 13:58:43.109184 1969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28b234ec8a5467eb43c194f41680db4" candidate_uuid: "f485fd5312ae46348cd6a664484b4510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dec3586cf4b4bf9801dee83df27167b" is_pre_election: true
I20240503 13:58:43.109184 2231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28b234ec8a5467eb43c194f41680db4" candidate_uuid: "f485fd5312ae46348cd6a664484b4510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff84bd968a714ba8bd790788b7f906ae" is_pre_election: true
I20240503 13:58:43.109506 1969 raft_consensus.cc:2454] T c28b234ec8a5467eb43c194f41680db4 P 5dec3586cf4b4bf9801dee83df27167b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f485fd5312ae46348cd6a664484b4510 in term 0.
I20240503 13:58:43.109505 2231 raft_consensus.cc:2454] T c28b234ec8a5467eb43c194f41680db4 P ff84bd968a714ba8bd790788b7f906ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f485fd5312ae46348cd6a664484b4510 in term 0.
I20240503 13:58:43.109918 2036 leader_election.cc:302] T c28b234ec8a5467eb43c194f41680db4 P f485fd5312ae46348cd6a664484b4510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f485fd5312ae46348cd6a664484b4510, ff84bd968a714ba8bd790788b7f906ae; no voters:
I20240503 13:58:43.110154 2563 raft_consensus.cc:2781] T c28b234ec8a5467eb43c194f41680db4 P f485fd5312ae46348cd6a664484b4510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:43.110221 2563 raft_consensus.cc:482] T c28b234ec8a5467eb43c194f41680db4 P f485fd5312ae46348cd6a664484b4510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:43.110251 2563 raft_consensus.cc:3037] T c28b234ec8a5467eb43c194f41680db4 P f485fd5312ae46348cd6a664484b4510 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:43.111127 2563 raft_consensus.cc:504] T c28b234ec8a5467eb43c194f41680db4 P f485fd5312ae46348cd6a664484b4510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dec3586cf4b4bf9801dee83df27167b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41733 } } peers { permanent_uuid: "f485fd5312ae46348cd6a664484b4510" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36393 } } peers { permanent_uuid: "ff84bd968a714ba8bd790788b7f906ae" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38155 } }
I20240503 13:58:43.111538 2563 leader_election.cc:288] T c28b234ec8a5467eb43c194f41680db4 P f485fd5312ae46348cd6a664484b4510 [CANDIDATE]: Term 1 election: Requested vote from peers 5dec3586cf4b4bf9801dee83df27167b (127.0.99.129:41733), ff84bd968a714ba8bd790788b7f906ae (127.0.99.131:38155)
I20240503 13:58:43.111847 1969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28b234ec8a5467eb43c194f41680db4" candidate_uuid: "f485fd5312ae46348cd6a664484b4510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dec3586cf4b4bf9801dee83df27167b"
I20240503 13:58:43.111901 2231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28b234ec8a5467eb43c194f41680db4" candidate_uuid: "f485fd5312ae46348cd6a664484b4510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff84bd968a714ba8bd790788b7f906ae"
I20240503 13:58:43.111992 2231 raft_consensus.cc:3037] T c28b234ec8a5467eb43c194f41680db4 P ff84bd968a714ba8bd790788b7f906ae [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:43.111989 1969 raft_consensus.cc:3037] T c28b234ec8a5467eb43c194f41680db4 P 5dec3586cf4b4bf9801dee83df27167b [term 0 FOLLOWER]: Advancing to term 1
W20240503 13:58:43.112193 2278 tablet.cc:2369] T c28b234ec8a5467eb43c194f41680db4 P ff84bd968a714ba8bd790788b7f906ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:58:43.113196 1969 raft_consensus.cc:2454] T c28b234ec8a5467eb43c194f41680db4 P 5dec3586cf4b4bf9801dee83df27167b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f485fd5312ae46348cd6a664484b4510 in term 1.
I20240503 13:58:43.113520 2035 leader_election.cc:302] T c28b234ec8a5467eb43c194f41680db4 P f485fd5312ae46348cd6a664484b4510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dec3586cf4b4bf9801dee83df27167b, f485fd5312ae46348cd6a664484b4510; no voters:
I20240503 13:58:43.113744 2563 raft_consensus.cc:2781] T c28b234ec8a5467eb43c194f41680db4 P f485fd5312ae46348cd6a664484b4510 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:43.113909 2231 raft_consensus.cc:2454] T c28b234ec8a5467eb43c194f41680db4 P ff84bd968a714ba8bd790788b7f906ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f485fd5312ae46348cd6a664484b4510 in term 1.
I20240503 13:58:43.114004 2563 raft_consensus.cc:686] T c28b234ec8a5467eb43c194f41680db4 P f485fd5312ae46348cd6a664484b4510 [term 1 LEADER]: Becoming Leader. State: Replica: f485fd5312ae46348cd6a664484b4510, State: Running, Role: LEADER
I20240503 13:58:43.114751 2563 consensus_queue.cc:237] T c28b234ec8a5467eb43c194f41680db4 P f485fd5312ae46348cd6a664484b4510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dec3586cf4b4bf9801dee83df27167b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41733 } } peers { permanent_uuid: "f485fd5312ae46348cd6a664484b4510" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36393 } } peers { permanent_uuid: "ff84bd968a714ba8bd790788b7f906ae" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38155 } }
I20240503 13:58:43.118364 1825 catalog_manager.cc:5484] T c28b234ec8a5467eb43c194f41680db4 P f485fd5312ae46348cd6a664484b4510 reported cstate change: term changed from 0 to 1, leader changed from <none> to f485fd5312ae46348cd6a664484b4510 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "f485fd5312ae46348cd6a664484b4510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dec3586cf4b4bf9801dee83df27167b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f485fd5312ae46348cd6a664484b4510" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff84bd968a714ba8bd790788b7f906ae" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38155 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:43.146872 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 13:58:43.148252 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 5dec3586cf4b4bf9801dee83df27167b to finish bootstrapping
I20240503 13:58:43.154173 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f485fd5312ae46348cd6a664484b4510 to finish bootstrapping
I20240503 13:58:43.159763 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ff84bd968a714ba8bd790788b7f906ae to finish bootstrapping
I20240503 13:58:43.165964 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 2d842902ff4d471581aef2afd45e74f5 to finish bootstrapping
I20240503 13:58:43.171558 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 7823d31eecb64f8ba2b8094cb0db8132 to finish bootstrapping
I20240503 13:58:43.176798 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 1887
W20240503 13:58:43.181370 2147 tablet.cc:2369] T c28b234ec8a5467eb43c194f41680db4 P f485fd5312ae46348cd6a664484b4510: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:58:43.371138 2342 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:43.371780 2473 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:43.372707 2080 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:58:43.374843 2211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:43.449326 2342 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:43.450057 2211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:43.450062 2080 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:58:43.450079 2473 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 13:58:43.525061 2035 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:41733: connect: Connection refused (error 111)
W20240503 13:58:43.525892 2035 consensus_peers.cc:485] T c28b234ec8a5467eb43c194f41680db4 P f485fd5312ae46348cd6a664484b4510 -> Peer 5dec3586cf4b4bf9801dee83df27167b (127.0.99.129:41733): Couldn't send request to peer 5dec3586cf4b4bf9801dee83df27167b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:41733: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:43.536316 2342 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:43.537406 2211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:43.537635 2473 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:43.537748 2080 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:58:43.674053 2563 consensus_queue.cc:1035] T c28b234ec8a5467eb43c194f41680db4 P f485fd5312ae46348cd6a664484b4510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff84bd968a714ba8bd790788b7f906ae" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:43.747243 2342 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:43.748023 2080 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:58:43.748350 2211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:43.749487 2473 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:43.793385 2408 heartbeater.cc:503] Master 127.0.99.190:33621 was elected leader, sending a full tablet report...
I20240503 13:58:43.822717 2342 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:43.822821 2211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:43.823446 2080 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:58:43.831853 2473 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:43.936643 2080 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:58:43.937228 2342 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:43.937229 2211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:43.937611 2473 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:43.972767 2539 heartbeater.cc:503] Master 127.0.99.190:33621 was elected leader, sending a full tablet report...
I20240503 13:58:43.973143 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 2018
I20240503 13:58:43.979709 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 2149
I20240503 13:58:43.988493 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 2280
I20240503 13:58:43.995170 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 2411
I20240503 13:58:44.001204 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 1796
2024-05-03T13:58:44Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/1 (2220 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2221 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/1
I20240503 13:58:44.024499 398 test_util.cc:268] Using random seed: 505932974
I20240503 13:58:44.025233 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 13:58:44.025278 398 ts_itest-base.cc:113] --------------
I20240503 13:58:44.025290 398 ts_itest-base.cc:114] 7 tablet servers
I20240503 13:58:44.025300 398 ts_itest-base.cc:115] 1 replicas per TS
I20240503 13:58:44.025310 398 ts_itest-base.cc:116] --------------
2024-05-03T13:58:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T13:58:44Z Disabled control of system clock
I20240503 13:58:44.032913 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:37103
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:37103
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 13:58:44.135759 2760 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 13:58:44.136025 2760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:44.136090 2760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:44.136165 2760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:44.139215 2760 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:44.139321 2760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 13:58:44.139376 2760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:44.139425 2760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 13:58:44.139462 2760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 13:58:44.143110 2760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:37103
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:37103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.2760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:44.144007 2760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:44.145066 2760 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:44.152501 2766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:44.153242 2765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:44.154026 2768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:44.154829 2760 server_base.cc:1034] running on GCE node
I20240503 13:58:44.155323 2760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:44.156208 2760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:44.157353 2760 hybrid_clock.cc:648] HybridClock initialized: now 1714744724157327 us; error 39 us; skew 500 ppm
I20240503 13:58:44.159201 2760 webserver.cc:460] Webserver started at http://127.0.99.190:34861/ using document root <none> and password file <none>
I20240503 13:58:44.159754 2760 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:44.159811 2760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:44.159998 2760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:44.161634 2760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b0ba17e1ed7e481fa58f7ce6b035ccaa"
format_stamp: "Formatted at 2024-05-03 13:58:44 on dist-test-slave-fr9h"
I20240503 13:58:44.162109 2760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b0ba17e1ed7e481fa58f7ce6b035ccaa"
format_stamp: "Formatted at 2024-05-03 13:58:44 on dist-test-slave-fr9h"
I20240503 13:58:44.165526 2760 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 13:58:44.167596 2774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:44.168567 2760 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:44.168669 2760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b0ba17e1ed7e481fa58f7ce6b035ccaa"
format_stamp: "Formatted at 2024-05-03 13:58:44 on dist-test-slave-fr9h"
I20240503 13:58:44.168762 2760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:44.194813 2760 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:44.195536 2760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:44.195698 2760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:44.202273 2760 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:37103
I20240503 13:58:44.202275 2826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:37103 every 8 connection(s)
I20240503 13:58:44.203235 2760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 13:58:44.205849 2827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:44.207993 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 2760
I20240503 13:58:44.208133 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 13:58:44.211339 2827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ba17e1ed7e481fa58f7ce6b035ccaa: Bootstrap starting.
I20240503 13:58:44.213802 2827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0ba17e1ed7e481fa58f7ce6b035ccaa: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:44.214619 2827 log.cc:824] T 00000000000000000000000000000000 P b0ba17e1ed7e481fa58f7ce6b035ccaa: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:44.216423 2827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ba17e1ed7e481fa58f7ce6b035ccaa: No bootstrap required, opened a new log
I20240503 13:58:44.218961 2827 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0ba17e1ed7e481fa58f7ce6b035ccaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ba17e1ed7e481fa58f7ce6b035ccaa" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37103 } }
I20240503 13:58:44.219133 2827 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0ba17e1ed7e481fa58f7ce6b035ccaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:44.219182 2827 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0ba17e1ed7e481fa58f7ce6b035ccaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ba17e1ed7e481fa58f7ce6b035ccaa, State: Initialized, Role: FOLLOWER
I20240503 13:58:44.219602 2827 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0ba17e1ed7e481fa58f7ce6b035ccaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ba17e1ed7e481fa58f7ce6b035ccaa" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37103 } }
I20240503 13:58:44.219718 2827 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0ba17e1ed7e481fa58f7ce6b035ccaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:58:44.219775 2827 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0ba17e1ed7e481fa58f7ce6b035ccaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:58:44.219883 2827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0ba17e1ed7e481fa58f7ce6b035ccaa [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:44.220716 2827 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0ba17e1ed7e481fa58f7ce6b035ccaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ba17e1ed7e481fa58f7ce6b035ccaa" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37103 } }
I20240503 13:58:44.221006 2827 leader_election.cc:302] T 00000000000000000000000000000000 P b0ba17e1ed7e481fa58f7ce6b035ccaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0ba17e1ed7e481fa58f7ce6b035ccaa; no voters:
I20240503 13:58:44.221222 2827 leader_election.cc:288] T 00000000000000000000000000000000 P b0ba17e1ed7e481fa58f7ce6b035ccaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:58:44.221329 2832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0ba17e1ed7e481fa58f7ce6b035ccaa [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:44.221873 2832 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0ba17e1ed7e481fa58f7ce6b035ccaa [term 1 LEADER]: Becoming Leader. State: Replica: b0ba17e1ed7e481fa58f7ce6b035ccaa, State: Running, Role: LEADER
I20240503 13:58:44.222149 2832 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0ba17e1ed7e481fa58f7ce6b035ccaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ba17e1ed7e481fa58f7ce6b035ccaa" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37103 } }
I20240503 13:58:44.222206 2827 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0ba17e1ed7e481fa58f7ce6b035ccaa [sys.catalog]: configured and running, proceeding with master startup.
I20240503 13:58:44.223505 2834 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ba17e1ed7e481fa58f7ce6b035ccaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0ba17e1ed7e481fa58f7ce6b035ccaa. Latest consensus state: current_term: 1 leader_uuid: "b0ba17e1ed7e481fa58f7ce6b035ccaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ba17e1ed7e481fa58f7ce6b035ccaa" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37103 } } }
I20240503 13:58:44.223541 2833 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ba17e1ed7e481fa58f7ce6b035ccaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0ba17e1ed7e481fa58f7ce6b035ccaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ba17e1ed7e481fa58f7ce6b035ccaa" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37103 } } }
I20240503 13:58:44.223610 2834 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ba17e1ed7e481fa58f7ce6b035ccaa [sys.catalog]: This master's current role is: LEADER
I20240503 13:58:44.223682 2833 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ba17e1ed7e481fa58f7ce6b035ccaa [sys.catalog]: This master's current role is: LEADER
I20240503 13:58:44.224069 2841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 13:58:44.226951 2841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 13:58:44.231984 2841 catalog_manager.cc:1347] Generated new cluster ID: 27fe641d049e423d9c8a8f478c28b7dc
I20240503 13:58:44.232041 2841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 13:58:44.242406 2841 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 13:58:44.243256 2841 catalog_manager.cc:1504] Loading token signing keys...
I20240503 13:58:44.250977 2841 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b0ba17e1ed7e481fa58f7ce6b035ccaa: Generated new TSK 0
I20240503 13:58:44.251773 2841 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 13:58:44.258534 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37103
--builtin_ntp_servers=127.0.99.148:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 13:58:44.368007 2851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:44.368209 2851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:44.368252 2851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:44.371212 2851 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:44.371269 2851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:44.371404 2851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 13:58:44.374966 2851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37103
--never_fsync=true
--heap_profile_path=/tmp/kudu.2851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:44.375823 2851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:44.376935 2851 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:44.386135 2856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:44.386831 2859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:44.387647 2857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:44.388502 2851 server_base.cc:1034] running on GCE node
I20240503 13:58:44.388870 2851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:44.389425 2851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:44.390587 2851 hybrid_clock.cc:648] HybridClock initialized: now 1714744724390568 us; error 38 us; skew 500 ppm
I20240503 13:58:44.392513 2851 webserver.cc:460] Webserver started at http://127.0.99.129:35957/ using document root <none> and password file <none>
I20240503 13:58:44.393051 2851 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:44.393111 2851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:44.393282 2851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:44.394987 2851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a9c26918b0724416878324492440c93d"
format_stamp: "Formatted at 2024-05-03 13:58:44 on dist-test-slave-fr9h"
I20240503 13:58:44.395483 2851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a9c26918b0724416878324492440c93d"
format_stamp: "Formatted at 2024-05-03 13:58:44 on dist-test-slave-fr9h"
I20240503 13:58:44.398669 2851 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240503 13:58:44.400888 2865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:44.401815 2851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:44.401924 2851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a9c26918b0724416878324492440c93d"
format_stamp: "Formatted at 2024-05-03 13:58:44 on dist-test-slave-fr9h"
I20240503 13:58:44.402002 2851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:44.433274 2851 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:44.433964 2851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:44.434128 2851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:44.434684 2851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:44.435621 2851 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:44.435730 2851 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:44.435839 2851 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:44.435875 2851 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:44.444936 2851 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:39461
I20240503 13:58:44.444958 2978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:39461 every 8 connection(s)
I20240503 13:58:44.445876 2851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 13:58:44.454506 2979 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37103
I20240503 13:58:44.454715 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 2851
I20240503 13:58:44.454746 2979 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:44.454808 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 13:58:44.455349 2979 heartbeater.cc:511] Master 127.0.99.190:37103 requested a full tablet report, sending...
I20240503 13:58:44.456934 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37103
--builtin_ntp_servers=127.0.99.148:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:58:44.456998 2791 ts_manager.cc:194] Registered new tserver with Master: a9c26918b0724416878324492440c93d (127.0.99.129:39461)
I20240503 13:58:44.458537 2791 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:38219
W20240503 13:58:44.558065 2982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:44.558247 2982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:44.558284 2982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:44.561043 2982 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:44.561097 2982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:44.561161 2982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 13:58:44.564468 2982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37103
--never_fsync=true
--heap_profile_path=/tmp/kudu.2982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:44.565236 2982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:44.566411 2982 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:44.575671 2987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:44.576526 2988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:44.577266 2990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:44.578083 2982 server_base.cc:1034] running on GCE node
I20240503 13:58:44.578369 2982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:44.578833 2982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:44.579981 2982 hybrid_clock.cc:648] HybridClock initialized: now 1714744724579962 us; error 31 us; skew 500 ppm
I20240503 13:58:44.581863 2982 webserver.cc:460] Webserver started at http://127.0.99.130:34345/ using document root <none> and password file <none>
I20240503 13:58:44.582358 2982 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:44.582417 2982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:44.582579 2982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:44.584152 2982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "694cd43f32a746e7ab0e91db5785ba52"
format_stamp: "Formatted at 2024-05-03 13:58:44 on dist-test-slave-fr9h"
I20240503 13:58:44.584604 2982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "694cd43f32a746e7ab0e91db5785ba52"
format_stamp: "Formatted at 2024-05-03 13:58:44 on dist-test-slave-fr9h"
I20240503 13:58:44.587769 2982 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 13:58:44.589818 2996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:44.590785 2982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:44.590896 2982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "694cd43f32a746e7ab0e91db5785ba52"
format_stamp: "Formatted at 2024-05-03 13:58:44 on dist-test-slave-fr9h"
I20240503 13:58:44.590974 2982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:44.607573 2982 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:44.608134 2982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:44.608278 2982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:44.608752 2982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:44.609529 2982 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:44.609580 2982 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:44.609630 2982 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:44.609661 2982 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:44.617491 2982 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:34403
I20240503 13:58:44.617537 3109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:34403 every 8 connection(s)
I20240503 13:58:44.618305 2982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 13:58:44.622673 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 2982
I20240503 13:58:44.622836 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 13:58:44.624986 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37103
--builtin_ntp_servers=127.0.99.148:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:58:44.627765 3110 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37103
I20240503 13:58:44.627995 3110 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:44.628612 3110 heartbeater.cc:511] Master 127.0.99.190:37103 requested a full tablet report, sending...
I20240503 13:58:44.629604 2791 ts_manager.cc:194] Registered new tserver with Master: 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130:34403)
I20240503 13:58:44.630117 2791 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:56379
W20240503 13:58:44.727514 3113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:44.727705 3113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:44.727756 3113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:44.730710 3113 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:44.730773 3113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:44.730876 3113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 13:58:44.734309 3113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37103
--never_fsync=true
--heap_profile_path=/tmp/kudu.3113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:44.735132 3113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:44.736213 3113 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:44.744506 3121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:44.745280 3118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:44.746069 3119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:44.746868 3113 server_base.cc:1034] running on GCE node
I20240503 13:58:44.747182 3113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:44.747714 3113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:44.748871 3113 hybrid_clock.cc:648] HybridClock initialized: now 1714744724748839 us; error 42 us; skew 500 ppm
I20240503 13:58:44.750746 3113 webserver.cc:460] Webserver started at http://127.0.99.131:34273/ using document root <none> and password file <none>
I20240503 13:58:44.751322 3113 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:44.751376 3113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:44.751545 3113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:44.753139 3113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9b3cf5210e6b4f2188e66a80dca823e6"
format_stamp: "Formatted at 2024-05-03 13:58:44 on dist-test-slave-fr9h"
I20240503 13:58:44.753614 3113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9b3cf5210e6b4f2188e66a80dca823e6"
format_stamp: "Formatted at 2024-05-03 13:58:44 on dist-test-slave-fr9h"
I20240503 13:58:44.756754 3113 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 13:58:44.758749 3127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:44.759714 3113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:44.759824 3113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9b3cf5210e6b4f2188e66a80dca823e6"
format_stamp: "Formatted at 2024-05-03 13:58:44 on dist-test-slave-fr9h"
I20240503 13:58:44.759904 3113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:44.781375 3113 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:44.781948 3113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:44.782099 3113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:44.782588 3113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:44.783418 3113 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:44.783463 3113 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:44.783519 3113 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:44.783551 3113 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:44.791525 3113 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:44145
I20240503 13:58:44.791584 3240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:44145 every 8 connection(s)
I20240503 13:58:44.792374 3113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 13:58:44.800670 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 3113
I20240503 13:58:44.800770 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 13:58:44.801024 3241 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37103
I20240503 13:58:44.801287 3241 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:44.801887 3241 heartbeater.cc:511] Master 127.0.99.190:37103 requested a full tablet report, sending...
I20240503 13:58:44.802801 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37103
--builtin_ntp_servers=127.0.99.148:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:58:44.802865 2791 ts_manager.cc:194] Registered new tserver with Master: 9b3cf5210e6b4f2188e66a80dca823e6 (127.0.99.131:44145)
I20240503 13:58:44.803437 2791 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:37997
W20240503 13:58:44.902675 3244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:44.902868 3244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:44.902933 3244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:44.905738 3244 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:44.905800 3244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:44.905877 3244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 13:58:44.909222 3244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37103
--never_fsync=true
--heap_profile_path=/tmp/kudu.3244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:44.909986 3244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:44.911010 3244 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:44.919026 3250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:44.919744 3249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:44.920590 3252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:44.921366 3244 server_base.cc:1034] running on GCE node
I20240503 13:58:44.921717 3244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:44.922190 3244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:44.923337 3244 hybrid_clock.cc:648] HybridClock initialized: now 1714744724923317 us; error 34 us; skew 500 ppm
I20240503 13:58:44.925160 3244 webserver.cc:460] Webserver started at http://127.0.99.132:42019/ using document root <none> and password file <none>
I20240503 13:58:44.925673 3244 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:44.925732 3244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:44.925890 3244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:44.927402 3244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e488cf84f7b74419a218d3512e237fea"
format_stamp: "Formatted at 2024-05-03 13:58:44 on dist-test-slave-fr9h"
I20240503 13:58:44.927816 3244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e488cf84f7b74419a218d3512e237fea"
format_stamp: "Formatted at 2024-05-03 13:58:44 on dist-test-slave-fr9h"
I20240503 13:58:44.930670 3244 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 13:58:44.932644 3258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:44.933548 3244 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:44.933655 3244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e488cf84f7b74419a218d3512e237fea"
format_stamp: "Formatted at 2024-05-03 13:58:44 on dist-test-slave-fr9h"
I20240503 13:58:44.933730 3244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:44.946506 3244 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:44.947033 3244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:44.947173 3244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:44.947678 3244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:44.948482 3244 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:44.948527 3244 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:44.948594 3244 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:44.948621 3244 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:44.956781 3244 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:40887
I20240503 13:58:44.956835 3371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:40887 every 8 connection(s)
I20240503 13:58:44.957652 3244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 13:58:44.965832 3372 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37103
I20240503 13:58:44.966020 3372 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:44.966465 3372 heartbeater.cc:511] Master 127.0.99.190:37103 requested a full tablet report, sending...
I20240503 13:58:44.967381 2791 ts_manager.cc:194] Registered new tserver with Master: e488cf84f7b74419a218d3512e237fea (127.0.99.132:40887)
I20240503 13:58:44.967732 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 3244
I20240503 13:58:44.967852 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 13:58:44.967949 2791 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:49065
I20240503 13:58:44.969787 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37103
--builtin_ntp_servers=127.0.99.148:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 13:58:45.069563 3375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:45.069731 3375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:45.069766 3375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:45.072739 3375 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:45.072796 3375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:45.072883 3375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 13:58:45.076279 3375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37103
--never_fsync=true
--heap_profile_path=/tmp/kudu.3375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:45.077054 3375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:45.078092 3375 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:45.086205 3380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:45.086977 3381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:45.087735 3383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:45.088533 3375 server_base.cc:1034] running on GCE node
I20240503 13:58:45.088888 3375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:45.089367 3375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:45.090540 3375 hybrid_clock.cc:648] HybridClock initialized: now 1714744725090524 us; error 28 us; skew 500 ppm
I20240503 13:58:45.092224 3375 webserver.cc:460] Webserver started at http://127.0.99.133:45247/ using document root <none> and password file <none>
I20240503 13:58:45.092708 3375 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:45.092762 3375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:45.092914 3375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:45.094429 3375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "acc5e87f231a46f5beb89cc24cbe28c4"
format_stamp: "Formatted at 2024-05-03 13:58:45 on dist-test-slave-fr9h"
I20240503 13:58:45.094906 3375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "acc5e87f231a46f5beb89cc24cbe28c4"
format_stamp: "Formatted at 2024-05-03 13:58:45 on dist-test-slave-fr9h"
I20240503 13:58:45.097899 3375 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 13:58:45.099758 3389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:45.100697 3375 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:45.100805 3375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "acc5e87f231a46f5beb89cc24cbe28c4"
format_stamp: "Formatted at 2024-05-03 13:58:45 on dist-test-slave-fr9h"
I20240503 13:58:45.100880 3375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:45.133155 3375 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:45.133819 3375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:45.133976 3375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:45.134487 3375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:45.135381 3375 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:45.135429 3375 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:45.135483 3375 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:45.135511 3375 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:45.143520 3375 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:45407
I20240503 13:58:45.143561 3502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:45407 every 8 connection(s)
I20240503 13:58:45.144356 3375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 13:58:45.144773 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 3375
I20240503 13:58:45.144891 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 13:58:45.147039 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37103
--builtin_ntp_servers=127.0.99.148:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:58:45.153414 3503 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37103
I20240503 13:58:45.153611 3503 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:45.154062 3503 heartbeater.cc:511] Master 127.0.99.190:37103 requested a full tablet report, sending...
I20240503 13:58:45.154896 2791 ts_manager.cc:194] Registered new tserver with Master: acc5e87f231a46f5beb89cc24cbe28c4 (127.0.99.133:45407)
I20240503 13:58:45.155439 2791 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:54165
W20240503 13:58:45.252801 3506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:45.253024 3506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:45.253095 3506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:45.256116 3506 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:45.256182 3506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:45.256266 3506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 13:58:45.259769 3506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37103
--never_fsync=true
--heap_profile_path=/tmp/kudu.3506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:45.260624 3506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:45.261667 3506 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:45.270082 3514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:45.270876 3511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:45.271661 3512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:45.272507 3506 server_base.cc:1034] running on GCE node
I20240503 13:58:45.272830 3506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:45.273478 3506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:45.274646 3506 hybrid_clock.cc:648] HybridClock initialized: now 1714744725274621 us; error 41 us; skew 500 ppm
I20240503 13:58:45.276422 3506 webserver.cc:460] Webserver started at http://127.0.99.134:46313/ using document root <none> and password file <none>
I20240503 13:58:45.276966 3506 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:45.277021 3506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:45.277194 3506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:45.278709 3506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "3b1cfcd5138346b6ac5b8581c536034d"
format_stamp: "Formatted at 2024-05-03 13:58:45 on dist-test-slave-fr9h"
I20240503 13:58:45.279167 3506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "3b1cfcd5138346b6ac5b8581c536034d"
format_stamp: "Formatted at 2024-05-03 13:58:45 on dist-test-slave-fr9h"
I20240503 13:58:45.282266 3506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20240503 13:58:45.284432 3520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:45.285379 3506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:45.285475 3506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "3b1cfcd5138346b6ac5b8581c536034d"
format_stamp: "Formatted at 2024-05-03 13:58:45 on dist-test-slave-fr9h"
I20240503 13:58:45.285573 3506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:45.307808 3506 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:45.308383 3506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:45.308529 3506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:45.309043 3506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:45.309861 3506 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:45.309906 3506 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:45.309949 3506 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:45.309979 3506 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:45.318200 3506 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:38729
I20240503 13:58:45.318233 3633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:38729 every 8 connection(s)
I20240503 13:58:45.319058 3506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 13:58:45.322527 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 3506
I20240503 13:58:45.322650 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 13:58:45.324734 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37103
--builtin_ntp_servers=127.0.99.148:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:58:45.327600 3634 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37103
I20240503 13:58:45.327809 3634 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:45.328282 3634 heartbeater.cc:511] Master 127.0.99.190:37103 requested a full tablet report, sending...
I20240503 13:58:45.329135 2791 ts_manager.cc:194] Registered new tserver with Master: 3b1cfcd5138346b6ac5b8581c536034d (127.0.99.134:38729)
I20240503 13:58:45.329667 2791 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:39603
W20240503 13:58:45.427927 3637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:45.428122 3637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:45.428193 3637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:45.431138 3637 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:45.431203 3637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:45.431334 3637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 13:58:45.434824 3637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37103
--never_fsync=true
--heap_profile_path=/tmp/kudu.3637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:45.435631 3637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:45.436681 3637 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:45.445021 3642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:45.445850 3645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:45.446595 3643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:45.447441 3637 server_base.cc:1034] running on GCE node
I20240503 13:58:45.447786 3637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:45.448288 3637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:45.449457 3637 hybrid_clock.cc:648] HybridClock initialized: now 1714744725449442 us; error 34 us; skew 500 ppm
I20240503 13:58:45.451370 3637 webserver.cc:460] Webserver started at http://127.0.99.135:46583/ using document root <none> and password file <none>
I20240503 13:58:45.451910 3637 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:45.451969 3637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:45.452136 3637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:45.453627 3637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "5c33992e399d4e63a2cc07cb4a479316"
format_stamp: "Formatted at 2024-05-03 13:58:45 on dist-test-slave-fr9h"
I20240503 13:58:45.454063 3637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "5c33992e399d4e63a2cc07cb4a479316"
format_stamp: "Formatted at 2024-05-03 13:58:45 on dist-test-slave-fr9h"
I20240503 13:58:45.457062 3637 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 13:58:45.459116 3651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:45.460119 3637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:45.460214 3637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "5c33992e399d4e63a2cc07cb4a479316"
format_stamp: "Formatted at 2024-05-03 13:58:45 on dist-test-slave-fr9h"
I20240503 13:58:45.460289 3637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:45.461050 2979 heartbeater.cc:503] Master 127.0.99.190:37103 was elected leader, sending a full tablet report...
I20240503 13:58:45.480757 3637 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:45.481319 3637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:45.481462 3637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:45.481961 3637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:45.482751 3637 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:45.482797 3637 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:45.482854 3637 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:45.482882 3637 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:45.491529 3637 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:33161
I20240503 13:58:45.491559 3764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:33161 every 8 connection(s)
I20240503 13:58:45.492415 3637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 13:58:45.500175 3765 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37103
I20240503 13:58:45.500389 3765 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:45.500496 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 3637
I20240503 13:58:45.500613 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 13:58:45.500911 3765 heartbeater.cc:511] Master 127.0.99.190:37103 requested a full tablet report, sending...
I20240503 13:58:45.501861 2791 ts_manager.cc:194] Registered new tserver with Master: 5c33992e399d4e63a2cc07cb4a479316 (127.0.99.135:33161)
I20240503 13:58:45.502579 2791 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:41591
I20240503 13:58:45.502894 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 13:58:45.511181 2791 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44468:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 13:58:45.526721 3044 tablet_service.cc:1467] Processing CreateTablet for tablet f6f3c78225a94967973739fe8bd5a3c4 (DEFAULT_TABLE table=TestTable [id=43c917e4db244005b57bdc88699adddb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:45.527581 3044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6f3c78225a94967973739fe8bd5a3c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:45.532406 3779 tablet_bootstrap.cc:492] T f6f3c78225a94967973739fe8bd5a3c4 P 694cd43f32a746e7ab0e91db5785ba52: Bootstrap starting.
I20240503 13:58:45.534335 3779 tablet_bootstrap.cc:654] T f6f3c78225a94967973739fe8bd5a3c4 P 694cd43f32a746e7ab0e91db5785ba52: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:45.534992 3779 log.cc:824] T f6f3c78225a94967973739fe8bd5a3c4 P 694cd43f32a746e7ab0e91db5785ba52: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:45.536520 3779 tablet_bootstrap.cc:492] T f6f3c78225a94967973739fe8bd5a3c4 P 694cd43f32a746e7ab0e91db5785ba52: No bootstrap required, opened a new log
I20240503 13:58:45.536695 3779 ts_tablet_manager.cc:1367] T f6f3c78225a94967973739fe8bd5a3c4 P 694cd43f32a746e7ab0e91db5785ba52: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 13:58:45.539474 3779 raft_consensus.cc:348] T f6f3c78225a94967973739fe8bd5a3c4 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } }
I20240503 13:58:45.539670 3779 raft_consensus.cc:374] T f6f3c78225a94967973739fe8bd5a3c4 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:45.539736 3779 raft_consensus.cc:729] T f6f3c78225a94967973739fe8bd5a3c4 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 694cd43f32a746e7ab0e91db5785ba52, State: Initialized, Role: FOLLOWER
I20240503 13:58:45.540125 3779 consensus_queue.cc:260] T f6f3c78225a94967973739fe8bd5a3c4 P 694cd43f32a746e7ab0e91db5785ba52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } }
I20240503 13:58:45.540241 3779 raft_consensus.cc:388] T f6f3c78225a94967973739fe8bd5a3c4 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:58:45.540304 3779 raft_consensus.cc:482] T f6f3c78225a94967973739fe8bd5a3c4 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:58:45.540407 3779 raft_consensus.cc:3037] T f6f3c78225a94967973739fe8bd5a3c4 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:45.541440 3779 raft_consensus.cc:504] T f6f3c78225a94967973739fe8bd5a3c4 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } }
I20240503 13:58:45.541776 3779 leader_election.cc:302] T f6f3c78225a94967973739fe8bd5a3c4 P 694cd43f32a746e7ab0e91db5785ba52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 694cd43f32a746e7ab0e91db5785ba52; no voters:
I20240503 13:58:45.542054 3779 leader_election.cc:288] T f6f3c78225a94967973739fe8bd5a3c4 P 694cd43f32a746e7ab0e91db5785ba52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:58:45.542151 3781 raft_consensus.cc:2781] T f6f3c78225a94967973739fe8bd5a3c4 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:45.542351 3781 raft_consensus.cc:686] T f6f3c78225a94967973739fe8bd5a3c4 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Becoming Leader. State: Replica: 694cd43f32a746e7ab0e91db5785ba52, State: Running, Role: LEADER
I20240503 13:58:45.542625 3781 consensus_queue.cc:237] T f6f3c78225a94967973739fe8bd5a3c4 P 694cd43f32a746e7ab0e91db5785ba52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } }
I20240503 13:58:45.542711 3110 heartbeater.cc:503] Master 127.0.99.190:37103 was elected leader, sending a full tablet report...
I20240503 13:58:45.542871 3779 ts_tablet_manager.cc:1398] T f6f3c78225a94967973739fe8bd5a3c4 P 694cd43f32a746e7ab0e91db5785ba52: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20240503 13:58:45.545644 2791 catalog_manager.cc:5484] T f6f3c78225a94967973739fe8bd5a3c4 P 694cd43f32a746e7ab0e91db5785ba52 reported cstate change: term changed from 0 to 1, leader changed from <none> to 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "694cd43f32a746e7ab0e91db5785ba52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } health_report { overall_health: HEALTHY } } }
I20240503 13:58:45.552096 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 13:58:45.553324 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a9c26918b0724416878324492440c93d to finish bootstrapping
I20240503 13:58:45.558761 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 694cd43f32a746e7ab0e91db5785ba52 to finish bootstrapping
I20240503 13:58:45.563113 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 9b3cf5210e6b4f2188e66a80dca823e6 to finish bootstrapping
I20240503 13:58:45.568285 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e488cf84f7b74419a218d3512e237fea to finish bootstrapping
I20240503 13:58:45.573385 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver acc5e87f231a46f5beb89cc24cbe28c4 to finish bootstrapping
I20240503 13:58:45.578331 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3b1cfcd5138346b6ac5b8581c536034d to finish bootstrapping
I20240503 13:58:45.583355 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 5c33992e399d4e63a2cc07cb4a479316 to finish bootstrapping
I20240503 13:58:45.588255 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 3113
I20240503 13:58:45.593845 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 3244
I20240503 13:58:45.599336 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 3375
I20240503 13:58:45.604568 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 3506
I20240503 13:58:45.609612 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 3637
I20240503 13:58:49.366361 2791 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44468:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:58:49.377432 2912 tablet_service.cc:1467] Processing CreateTablet for tablet 03e82ebc742b46ce98fc9ab599b08f46 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7602235178ee496990dab5c7d3ce3a4a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:49.377439 2913 tablet_service.cc:1467] Processing CreateTablet for tablet ce166608b56540d89bddc4919356ebe7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7602235178ee496990dab5c7d3ce3a4a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:49.377439 2911 tablet_service.cc:1467] Processing CreateTablet for tablet 1c36890ea27847caa128b7896559d89f (DEFAULT_TABLE table=rebalance_test_table_0 [id=7602235178ee496990dab5c7d3ce3a4a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:49.378414 2912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e82ebc742b46ce98fc9ab599b08f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:49.378794 2911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c36890ea27847caa128b7896559d89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:49.378979 2913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce166608b56540d89bddc4919356ebe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:49.383411 3799 tablet_bootstrap.cc:492] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d: Bootstrap starting.
I20240503 13:58:49.384932 3799 tablet_bootstrap.cc:654] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:49.385774 3799 log.cc:824] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:49.387089 3799 tablet_bootstrap.cc:492] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d: No bootstrap required, opened a new log
I20240503 13:58:49.387228 3799 ts_tablet_manager.cc:1367] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 13:58:49.389909 3799 raft_consensus.cc:348] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:49.390105 3799 raft_consensus.cc:374] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:49.390178 3799 raft_consensus.cc:729] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9c26918b0724416878324492440c93d, State: Initialized, Role: FOLLOWER
I20240503 13:58:49.390591 3799 consensus_queue.cc:260] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:49.390712 3799 raft_consensus.cc:388] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:58:49.390769 3799 raft_consensus.cc:482] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:58:49.390856 3799 raft_consensus.cc:3037] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:49.391556 3799 raft_consensus.cc:504] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:49.391840 3799 leader_election.cc:302] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9c26918b0724416878324492440c93d; no voters:
I20240503 13:58:49.392086 3799 leader_election.cc:288] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:58:49.392174 3801 raft_consensus.cc:2781] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:49.392369 3801 raft_consensus.cc:686] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Becoming Leader. State: Replica: a9c26918b0724416878324492440c93d, State: Running, Role: LEADER
I20240503 13:58:49.392684 3801 consensus_queue.cc:237] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:49.392808 3799 ts_tablet_manager.cc:1398] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20240503 13:58:49.393154 3799 tablet_bootstrap.cc:492] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d: Bootstrap starting.
I20240503 13:58:49.394045 3799 tablet_bootstrap.cc:654] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:49.395136 3799 tablet_bootstrap.cc:492] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d: No bootstrap required, opened a new log
I20240503 13:58:49.395222 3799 ts_tablet_manager.cc:1367] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 13:58:49.395419 2791 catalog_manager.cc:5484] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d reported cstate change: term changed from 0 to 1, leader changed from <none> to a9c26918b0724416878324492440c93d (127.0.99.129). New cstate: current_term: 1 leader_uuid: "a9c26918b0724416878324492440c93d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } health_report { overall_health: HEALTHY } } }
I20240503 13:58:49.395754 3799 raft_consensus.cc:348] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:49.395824 3799 raft_consensus.cc:374] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:49.395848 3799 raft_consensus.cc:729] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9c26918b0724416878324492440c93d, State: Initialized, Role: FOLLOWER
I20240503 13:58:49.395960 3799 consensus_queue.cc:260] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:49.396035 3799 raft_consensus.cc:388] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:58:49.396063 3799 raft_consensus.cc:482] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:58:49.396096 3799 raft_consensus.cc:3037] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:49.396755 3799 raft_consensus.cc:504] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:49.396896 3799 leader_election.cc:302] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9c26918b0724416878324492440c93d; no voters:
I20240503 13:58:49.396991 3799 leader_election.cc:288] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:58:49.397033 3801 raft_consensus.cc:2781] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:49.397130 3801 raft_consensus.cc:686] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Becoming Leader. State: Replica: a9c26918b0724416878324492440c93d, State: Running, Role: LEADER
I20240503 13:58:49.397200 3799 ts_tablet_manager.cc:1398] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 13:58:49.397241 3801 consensus_queue.cc:237] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:49.397362 3799 tablet_bootstrap.cc:492] T 1c36890ea27847caa128b7896559d89f P a9c26918b0724416878324492440c93d: Bootstrap starting.
I20240503 13:58:49.398315 3799 tablet_bootstrap.cc:654] T 1c36890ea27847caa128b7896559d89f P a9c26918b0724416878324492440c93d: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:49.398916 2791 catalog_manager.cc:5484] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d reported cstate change: term changed from 0 to 1, leader changed from <none> to a9c26918b0724416878324492440c93d (127.0.99.129). New cstate: current_term: 1 leader_uuid: "a9c26918b0724416878324492440c93d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } health_report { overall_health: HEALTHY } } }
I20240503 13:58:49.399411 3799 tablet_bootstrap.cc:492] T 1c36890ea27847caa128b7896559d89f P a9c26918b0724416878324492440c93d: No bootstrap required, opened a new log
I20240503 13:58:49.399483 3799 ts_tablet_manager.cc:1367] T 1c36890ea27847caa128b7896559d89f P a9c26918b0724416878324492440c93d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:49.399904 3799 raft_consensus.cc:348] T 1c36890ea27847caa128b7896559d89f P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:49.399983 3799 raft_consensus.cc:374] T 1c36890ea27847caa128b7896559d89f P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:49.400025 3799 raft_consensus.cc:729] T 1c36890ea27847caa128b7896559d89f P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9c26918b0724416878324492440c93d, State: Initialized, Role: FOLLOWER
I20240503 13:58:49.400144 3799 consensus_queue.cc:260] T 1c36890ea27847caa128b7896559d89f P a9c26918b0724416878324492440c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:49.400226 3799 raft_consensus.cc:388] T 1c36890ea27847caa128b7896559d89f P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:58:49.400266 3799 raft_consensus.cc:482] T 1c36890ea27847caa128b7896559d89f P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:58:49.400321 3799 raft_consensus.cc:3037] T 1c36890ea27847caa128b7896559d89f P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:49.401130 3799 raft_consensus.cc:504] T 1c36890ea27847caa128b7896559d89f P a9c26918b0724416878324492440c93d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:49.401234 3799 leader_election.cc:302] T 1c36890ea27847caa128b7896559d89f P a9c26918b0724416878324492440c93d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9c26918b0724416878324492440c93d; no voters:
I20240503 13:58:49.401305 3799 leader_election.cc:288] T 1c36890ea27847caa128b7896559d89f P a9c26918b0724416878324492440c93d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:58:49.401393 3802 raft_consensus.cc:2781] T 1c36890ea27847caa128b7896559d89f P a9c26918b0724416878324492440c93d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:49.401525 3802 raft_consensus.cc:686] T 1c36890ea27847caa128b7896559d89f P a9c26918b0724416878324492440c93d [term 1 LEADER]: Becoming Leader. State: Replica: a9c26918b0724416878324492440c93d, State: Running, Role: LEADER
I20240503 13:58:49.401527 3799 ts_tablet_manager.cc:1398] T 1c36890ea27847caa128b7896559d89f P a9c26918b0724416878324492440c93d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 13:58:49.401690 3802 consensus_queue.cc:237] T 1c36890ea27847caa128b7896559d89f P a9c26918b0724416878324492440c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:49.402912 2791 catalog_manager.cc:5484] T 1c36890ea27847caa128b7896559d89f P a9c26918b0724416878324492440c93d reported cstate change: term changed from 0 to 1, leader changed from <none> to a9c26918b0724416878324492440c93d (127.0.99.129). New cstate: current_term: 1 leader_uuid: "a9c26918b0724416878324492440c93d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } health_report { overall_health: HEALTHY } } }
W20240503 13:58:49.510923 3810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:49.511157 3810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:49.514120 3810 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 71.4674 ms
time per row: 0.0357337 ms
I20240503 13:58:49.628221 2791 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44468:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:58:49.633317 3043 tablet_service.cc:1467] Processing CreateTablet for tablet 1a88836924b04bfea8524184d061a19b (DEFAULT_TABLE table=rebalance_test_table_1 [id=a1c069f80f8f412494da8d1b96fa3129]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:49.633317 3042 tablet_service.cc:1467] Processing CreateTablet for tablet 0f599173c7624329af379768658f6041 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a1c069f80f8f412494da8d1b96fa3129]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:49.633322 3044 tablet_service.cc:1467] Processing CreateTablet for tablet 1a0e0ac43355469cbce23fc6a7bb0cfa (DEFAULT_TABLE table=rebalance_test_table_1 [id=a1c069f80f8f412494da8d1b96fa3129]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:49.633699 3043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a88836924b04bfea8524184d061a19b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:49.633834 3044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a0e0ac43355469cbce23fc6a7bb0cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:49.634068 3042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f599173c7624329af379768658f6041. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:49.635879 3842 tablet_bootstrap.cc:492] T 1a0e0ac43355469cbce23fc6a7bb0cfa P 694cd43f32a746e7ab0e91db5785ba52: Bootstrap starting.
I20240503 13:58:49.636639 3842 tablet_bootstrap.cc:654] T 1a0e0ac43355469cbce23fc6a7bb0cfa P 694cd43f32a746e7ab0e91db5785ba52: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:49.637504 3842 tablet_bootstrap.cc:492] T 1a0e0ac43355469cbce23fc6a7bb0cfa P 694cd43f32a746e7ab0e91db5785ba52: No bootstrap required, opened a new log
I20240503 13:58:49.637579 3842 ts_tablet_manager.cc:1367] T 1a0e0ac43355469cbce23fc6a7bb0cfa P 694cd43f32a746e7ab0e91db5785ba52: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:49.638000 3842 raft_consensus.cc:348] T 1a0e0ac43355469cbce23fc6a7bb0cfa P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } }
I20240503 13:58:49.638108 3842 raft_consensus.cc:374] T 1a0e0ac43355469cbce23fc6a7bb0cfa P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:49.638142 3842 raft_consensus.cc:729] T 1a0e0ac43355469cbce23fc6a7bb0cfa P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 694cd43f32a746e7ab0e91db5785ba52, State: Initialized, Role: FOLLOWER
I20240503 13:58:49.638288 3842 consensus_queue.cc:260] T 1a0e0ac43355469cbce23fc6a7bb0cfa P 694cd43f32a746e7ab0e91db5785ba52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } }
I20240503 13:58:49.638372 3842 raft_consensus.cc:388] T 1a0e0ac43355469cbce23fc6a7bb0cfa P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:58:49.638415 3842 raft_consensus.cc:482] T 1a0e0ac43355469cbce23fc6a7bb0cfa P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:58:49.638458 3842 raft_consensus.cc:3037] T 1a0e0ac43355469cbce23fc6a7bb0cfa P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:49.639130 3842 raft_consensus.cc:504] T 1a0e0ac43355469cbce23fc6a7bb0cfa P 694cd43f32a746e7ab0e91db5785ba52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } }
I20240503 13:58:49.639233 3842 leader_election.cc:302] T 1a0e0ac43355469cbce23fc6a7bb0cfa P 694cd43f32a746e7ab0e91db5785ba52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 694cd43f32a746e7ab0e91db5785ba52; no voters:
I20240503 13:58:49.639386 3842 leader_election.cc:288] T 1a0e0ac43355469cbce23fc6a7bb0cfa P 694cd43f32a746e7ab0e91db5785ba52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:58:49.639490 3844 raft_consensus.cc:2781] T 1a0e0ac43355469cbce23fc6a7bb0cfa P 694cd43f32a746e7ab0e91db5785ba52 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:49.639680 3842 ts_tablet_manager.cc:1398] T 1a0e0ac43355469cbce23fc6a7bb0cfa P 694cd43f32a746e7ab0e91db5785ba52: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:49.639684 3844 raft_consensus.cc:686] T 1a0e0ac43355469cbce23fc6a7bb0cfa P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Becoming Leader. State: Replica: 694cd43f32a746e7ab0e91db5785ba52, State: Running, Role: LEADER
I20240503 13:58:49.639837 3842 tablet_bootstrap.cc:492] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52: Bootstrap starting.
I20240503 13:58:49.639853 3844 consensus_queue.cc:237] T 1a0e0ac43355469cbce23fc6a7bb0cfa P 694cd43f32a746e7ab0e91db5785ba52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } }
I20240503 13:58:49.640559 3842 tablet_bootstrap.cc:654] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:49.641381 2791 catalog_manager.cc:5484] T 1a0e0ac43355469cbce23fc6a7bb0cfa P 694cd43f32a746e7ab0e91db5785ba52 reported cstate change: term changed from 0 to 1, leader changed from <none> to 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "694cd43f32a746e7ab0e91db5785ba52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } health_report { overall_health: HEALTHY } } }
I20240503 13:58:49.641568 3842 tablet_bootstrap.cc:492] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52: No bootstrap required, opened a new log
I20240503 13:58:49.641685 3842 ts_tablet_manager.cc:1367] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:49.642139 3842 raft_consensus.cc:348] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } }
I20240503 13:58:49.642215 3842 raft_consensus.cc:374] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:49.642248 3842 raft_consensus.cc:729] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 694cd43f32a746e7ab0e91db5785ba52, State: Initialized, Role: FOLLOWER
I20240503 13:58:49.642361 3842 consensus_queue.cc:260] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } }
I20240503 13:58:49.642441 3842 raft_consensus.cc:388] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:58:49.642495 3842 raft_consensus.cc:482] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:58:49.642553 3842 raft_consensus.cc:3037] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:49.643384 3842 raft_consensus.cc:504] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } }
I20240503 13:58:49.643499 3842 leader_election.cc:302] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 694cd43f32a746e7ab0e91db5785ba52; no voters:
I20240503 13:58:49.643581 3842 leader_election.cc:288] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:58:49.643615 3844 raft_consensus.cc:2781] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:49.643674 3844 raft_consensus.cc:686] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Becoming Leader. State: Replica: 694cd43f32a746e7ab0e91db5785ba52, State: Running, Role: LEADER
I20240503 13:58:49.643774 3842 ts_tablet_manager.cc:1398] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:49.643767 3844 consensus_queue.cc:237] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } }
I20240503 13:58:49.643997 3842 tablet_bootstrap.cc:492] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52: Bootstrap starting.
I20240503 13:58:49.644857 3842 tablet_bootstrap.cc:654] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:49.645287 2791 catalog_manager.cc:5484] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 reported cstate change: term changed from 0 to 1, leader changed from <none> to 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "694cd43f32a746e7ab0e91db5785ba52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } health_report { overall_health: HEALTHY } } }
I20240503 13:58:49.645912 3842 tablet_bootstrap.cc:492] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52: No bootstrap required, opened a new log
I20240503 13:58:49.645987 3842 ts_tablet_manager.cc:1367] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:49.646360 3842 raft_consensus.cc:348] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } }
I20240503 13:58:49.646425 3842 raft_consensus.cc:374] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:49.646442 3842 raft_consensus.cc:729] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 694cd43f32a746e7ab0e91db5785ba52, State: Initialized, Role: FOLLOWER
I20240503 13:58:49.646560 3842 consensus_queue.cc:260] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } }
I20240503 13:58:49.646637 3842 raft_consensus.cc:388] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:58:49.646677 3842 raft_consensus.cc:482] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:58:49.646718 3842 raft_consensus.cc:3037] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:49.647260 3842 raft_consensus.cc:504] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } }
I20240503 13:58:49.647395 3842 leader_election.cc:302] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 694cd43f32a746e7ab0e91db5785ba52; no voters:
I20240503 13:58:49.647478 3842 leader_election.cc:288] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:58:49.647531 3844 raft_consensus.cc:2781] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:49.647629 3844 raft_consensus.cc:686] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Becoming Leader. State: Replica: 694cd43f32a746e7ab0e91db5785ba52, State: Running, Role: LEADER
I20240503 13:58:49.647670 3842 ts_tablet_manager.cc:1398] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 13:58:49.647718 3844 consensus_queue.cc:237] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } }
I20240503 13:58:49.648747 2791 catalog_manager.cc:5484] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 reported cstate change: term changed from 0 to 1, leader changed from <none> to 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "694cd43f32a746e7ab0e91db5785ba52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } health_report { overall_health: HEALTHY } } }
W20240503 13:58:49.749132 3853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:49.749364 3853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:49.752172 3853 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 61.6505 ms
time per row: 0.0308253 ms
I20240503 13:58:49.851244 2790 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44468:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:58:49.856160 2913 tablet_service.cc:1467] Processing CreateTablet for tablet a08c24279c7748dd9b10ff92c6d68845 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0de733125f7b4a63a5162b2d584f7379]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:49.856194 2912 tablet_service.cc:1467] Processing CreateTablet for tablet 25a1559451b8472482ac6e482da41d28 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0de733125f7b4a63a5162b2d584f7379]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:49.856200 2911 tablet_service.cc:1467] Processing CreateTablet for tablet 558e844e3d344fcbb0e41cdf132c880c (DEFAULT_TABLE table=rebalance_test_table_2 [id=0de733125f7b4a63a5162b2d584f7379]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:49.856544 2913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a08c24279c7748dd9b10ff92c6d68845. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:49.856717 2912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25a1559451b8472482ac6e482da41d28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:49.856834 2911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 558e844e3d344fcbb0e41cdf132c880c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:49.858309 3799 tablet_bootstrap.cc:492] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d: Bootstrap starting.
I20240503 13:58:49.859038 3799 tablet_bootstrap.cc:654] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:49.859918 3799 tablet_bootstrap.cc:492] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d: No bootstrap required, opened a new log
I20240503 13:58:49.859994 3799 ts_tablet_manager.cc:1367] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:58:49.860469 3799 raft_consensus.cc:348] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:49.860564 3799 raft_consensus.cc:374] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:49.860597 3799 raft_consensus.cc:729] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9c26918b0724416878324492440c93d, State: Initialized, Role: FOLLOWER
I20240503 13:58:49.860721 3799 consensus_queue.cc:260] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:49.860795 3799 raft_consensus.cc:388] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:58:49.860837 3799 raft_consensus.cc:482] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:58:49.860889 3799 raft_consensus.cc:3037] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:49.861582 3799 raft_consensus.cc:504] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:49.861703 3799 leader_election.cc:302] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9c26918b0724416878324492440c93d; no voters:
I20240503 13:58:49.861826 3799 leader_election.cc:288] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:58:49.861871 3802 raft_consensus.cc:2781] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:49.861949 3802 raft_consensus.cc:686] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Becoming Leader. State: Replica: a9c26918b0724416878324492440c93d, State: Running, Role: LEADER
I20240503 13:58:49.862061 3799 ts_tablet_manager.cc:1398] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:58:49.862052 3802 consensus_queue.cc:237] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:49.862224 3799 tablet_bootstrap.cc:492] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d: Bootstrap starting.
I20240503 13:58:49.862951 3799 tablet_bootstrap.cc:654] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:49.863878 2790 catalog_manager.cc:5484] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d reported cstate change: term changed from 0 to 1, leader changed from <none> to a9c26918b0724416878324492440c93d (127.0.99.129). New cstate: current_term: 1 leader_uuid: "a9c26918b0724416878324492440c93d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } health_report { overall_health: HEALTHY } } }
I20240503 13:58:49.863983 3799 tablet_bootstrap.cc:492] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d: No bootstrap required, opened a new log
I20240503 13:58:49.864044 3799 ts_tablet_manager.cc:1367] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:58:49.864446 3799 raft_consensus.cc:348] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:49.864516 3799 raft_consensus.cc:374] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:49.864535 3799 raft_consensus.cc:729] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9c26918b0724416878324492440c93d, State: Initialized, Role: FOLLOWER
I20240503 13:58:49.864643 3799 consensus_queue.cc:260] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:49.864706 3799 raft_consensus.cc:388] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:58:49.864782 3799 raft_consensus.cc:482] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:58:49.864843 3799 raft_consensus.cc:3037] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:49.865674 3799 raft_consensus.cc:504] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:49.865805 3799 leader_election.cc:302] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9c26918b0724416878324492440c93d; no voters:
I20240503 13:58:49.865896 3799 leader_election.cc:288] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:58:49.865950 3803 raft_consensus.cc:2781] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:49.866047 3803 raft_consensus.cc:686] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [term 1 LEADER]: Becoming Leader. State: Replica: a9c26918b0724416878324492440c93d, State: Running, Role: LEADER
I20240503 13:58:49.866148 3799 ts_tablet_manager.cc:1398] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:58:49.866158 3803 consensus_queue.cc:237] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:49.866299 3799 tablet_bootstrap.cc:492] T a08c24279c7748dd9b10ff92c6d68845 P a9c26918b0724416878324492440c93d: Bootstrap starting.
I20240503 13:58:49.867020 3799 tablet_bootstrap.cc:654] T a08c24279c7748dd9b10ff92c6d68845 P a9c26918b0724416878324492440c93d: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:49.867409 2790 catalog_manager.cc:5484] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d reported cstate change: term changed from 0 to 1, leader changed from <none> to a9c26918b0724416878324492440c93d (127.0.99.129). New cstate: current_term: 1 leader_uuid: "a9c26918b0724416878324492440c93d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } health_report { overall_health: HEALTHY } } }
I20240503 13:58:49.867969 3799 tablet_bootstrap.cc:492] T a08c24279c7748dd9b10ff92c6d68845 P a9c26918b0724416878324492440c93d: No bootstrap required, opened a new log
I20240503 13:58:49.868052 3799 ts_tablet_manager.cc:1367] T a08c24279c7748dd9b10ff92c6d68845 P a9c26918b0724416878324492440c93d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:58:49.868448 3799 raft_consensus.cc:348] T a08c24279c7748dd9b10ff92c6d68845 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:49.868541 3799 raft_consensus.cc:374] T a08c24279c7748dd9b10ff92c6d68845 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:49.868582 3799 raft_consensus.cc:729] T a08c24279c7748dd9b10ff92c6d68845 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9c26918b0724416878324492440c93d, State: Initialized, Role: FOLLOWER
I20240503 13:58:49.868691 3799 consensus_queue.cc:260] T a08c24279c7748dd9b10ff92c6d68845 P a9c26918b0724416878324492440c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:49.868834 3799 raft_consensus.cc:388] T a08c24279c7748dd9b10ff92c6d68845 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:58:49.868880 3799 raft_consensus.cc:482] T a08c24279c7748dd9b10ff92c6d68845 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:58:49.868956 3799 raft_consensus.cc:3037] T a08c24279c7748dd9b10ff92c6d68845 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:49.869596 3799 raft_consensus.cc:504] T a08c24279c7748dd9b10ff92c6d68845 P a9c26918b0724416878324492440c93d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:49.869702 3799 leader_election.cc:302] T a08c24279c7748dd9b10ff92c6d68845 P a9c26918b0724416878324492440c93d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9c26918b0724416878324492440c93d; no voters:
I20240503 13:58:49.869805 3799 leader_election.cc:288] T a08c24279c7748dd9b10ff92c6d68845 P a9c26918b0724416878324492440c93d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:58:49.869841 3803 raft_consensus.cc:2781] T a08c24279c7748dd9b10ff92c6d68845 P a9c26918b0724416878324492440c93d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:49.869917 3803 raft_consensus.cc:686] T a08c24279c7748dd9b10ff92c6d68845 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Becoming Leader. State: Replica: a9c26918b0724416878324492440c93d, State: Running, Role: LEADER
I20240503 13:58:49.870013 3803 consensus_queue.cc:237] T a08c24279c7748dd9b10ff92c6d68845 P a9c26918b0724416878324492440c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:49.870083 3799 ts_tablet_manager.cc:1398] T a08c24279c7748dd9b10ff92c6d68845 P a9c26918b0724416878324492440c93d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:58:49.871016 2790 catalog_manager.cc:5484] T a08c24279c7748dd9b10ff92c6d68845 P a9c26918b0724416878324492440c93d reported cstate change: term changed from 0 to 1, leader changed from <none> to a9c26918b0724416878324492440c93d (127.0.99.129). New cstate: current_term: 1 leader_uuid: "a9c26918b0724416878324492440c93d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } health_report { overall_health: HEALTHY } } }
W20240503 13:58:49.974285 3891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:49.974509 3891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:49.977316 3891 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 64.3107 ms
time per row: 0.0321553 ms
I20240503 13:58:50.079123 2784 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44468:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:58:50.084118 3043 tablet_service.cc:1467] Processing CreateTablet for tablet d140fb732825493dbcd7e41ea7e62271 (DEFAULT_TABLE table=rebalance_test_table_3 [id=785472d7e43343c0a92a6afb07b14c83]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:50.084120 3042 tablet_service.cc:1467] Processing CreateTablet for tablet d899ba5a20f0466da4e0e3fc9831e5f5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=785472d7e43343c0a92a6afb07b14c83]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:50.084187 3044 tablet_service.cc:1467] Processing CreateTablet for tablet fa9ffe2aeedf42e89d4e00ac12fb9e92 (DEFAULT_TABLE table=rebalance_test_table_3 [id=785472d7e43343c0a92a6afb07b14c83]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:50.084483 3043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d140fb732825493dbcd7e41ea7e62271. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:50.086205 3044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa9ffe2aeedf42e89d4e00ac12fb9e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:50.086243 3842 tablet_bootstrap.cc:492] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52: Bootstrap starting.
I20240503 13:58:50.086421 3042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d899ba5a20f0466da4e0e3fc9831e5f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:50.087098 3842 tablet_bootstrap.cc:654] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:50.088398 3842 tablet_bootstrap.cc:492] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52: No bootstrap required, opened a new log
I20240503 13:58:50.088485 3842 ts_tablet_manager.cc:1367] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:58:50.088850 3842 raft_consensus.cc:348] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } }
I20240503 13:58:50.088927 3842 raft_consensus.cc:374] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:50.088961 3842 raft_consensus.cc:729] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 694cd43f32a746e7ab0e91db5785ba52, State: Initialized, Role: FOLLOWER
I20240503 13:58:50.089087 3842 consensus_queue.cc:260] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } }
I20240503 13:58:50.089149 3842 raft_consensus.cc:388] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:58:50.089183 3842 raft_consensus.cc:482] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:58:50.089236 3842 raft_consensus.cc:3037] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:50.089810 3842 raft_consensus.cc:504] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } }
I20240503 13:58:50.089923 3842 leader_election.cc:302] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 694cd43f32a746e7ab0e91db5785ba52; no voters:
I20240503 13:58:50.090013 3842 leader_election.cc:288] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:58:50.090054 3844 raft_consensus.cc:2781] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:50.090152 3844 raft_consensus.cc:686] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Becoming Leader. State: Replica: 694cd43f32a746e7ab0e91db5785ba52, State: Running, Role: LEADER
I20240503 13:58:50.090260 3844 consensus_queue.cc:237] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } }
I20240503 13:58:50.090405 3842 ts_tablet_manager.cc:1398] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 13:58:50.090546 3842 tablet_bootstrap.cc:492] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52: Bootstrap starting.
I20240503 13:58:50.091552 2784 catalog_manager.cc:5484] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 reported cstate change: term changed from 0 to 1, leader changed from <none> to 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "694cd43f32a746e7ab0e91db5785ba52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } health_report { overall_health: HEALTHY } } }
I20240503 13:58:50.091723 3842 tablet_bootstrap.cc:654] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:50.092655 3842 tablet_bootstrap.cc:492] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52: No bootstrap required, opened a new log
I20240503 13:58:50.092736 3842 ts_tablet_manager.cc:1367] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:50.093071 3842 raft_consensus.cc:348] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } }
I20240503 13:58:50.093178 3842 raft_consensus.cc:374] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:50.093226 3842 raft_consensus.cc:729] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 694cd43f32a746e7ab0e91db5785ba52, State: Initialized, Role: FOLLOWER
I20240503 13:58:50.093338 3842 consensus_queue.cc:260] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } }
I20240503 13:58:50.093405 3842 raft_consensus.cc:388] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:58:50.093444 3842 raft_consensus.cc:482] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:58:50.093497 3842 raft_consensus.cc:3037] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:50.094334 3842 raft_consensus.cc:504] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } }
I20240503 13:58:50.094461 3842 leader_election.cc:302] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 694cd43f32a746e7ab0e91db5785ba52; no voters:
I20240503 13:58:50.094559 3842 leader_election.cc:288] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:58:50.094620 3844 raft_consensus.cc:2781] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:50.094715 3844 raft_consensus.cc:686] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Becoming Leader. State: Replica: 694cd43f32a746e7ab0e91db5785ba52, State: Running, Role: LEADER
I20240503 13:58:50.094831 3842 ts_tablet_manager.cc:1398] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:50.094815 3844 consensus_queue.cc:237] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } }
I20240503 13:58:50.094970 3842 tablet_bootstrap.cc:492] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52: Bootstrap starting.
I20240503 13:58:50.095772 3842 tablet_bootstrap.cc:654] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:50.095989 2787 catalog_manager.cc:5484] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 reported cstate change: term changed from 0 to 1, leader changed from <none> to 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "694cd43f32a746e7ab0e91db5785ba52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } health_report { overall_health: HEALTHY } } }
I20240503 13:58:50.097008 3842 tablet_bootstrap.cc:492] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52: No bootstrap required, opened a new log
I20240503 13:58:50.097079 3842 ts_tablet_manager.cc:1367] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:58:50.097393 3842 raft_consensus.cc:348] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } }
I20240503 13:58:50.097460 3842 raft_consensus.cc:374] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:50.097481 3842 raft_consensus.cc:729] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 694cd43f32a746e7ab0e91db5785ba52, State: Initialized, Role: FOLLOWER
I20240503 13:58:50.097604 3842 consensus_queue.cc:260] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } }
I20240503 13:58:50.097688 3842 raft_consensus.cc:388] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:58:50.097733 3842 raft_consensus.cc:482] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:58:50.097776 3842 raft_consensus.cc:3037] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:50.098456 3842 raft_consensus.cc:504] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } }
I20240503 13:58:50.098551 3842 leader_election.cc:302] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 694cd43f32a746e7ab0e91db5785ba52; no voters:
I20240503 13:58:50.098642 3842 leader_election.cc:288] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:58:50.098695 3845 raft_consensus.cc:2781] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:50.098781 3845 raft_consensus.cc:686] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Becoming Leader. State: Replica: 694cd43f32a746e7ab0e91db5785ba52, State: Running, Role: LEADER
I20240503 13:58:50.098865 3842 ts_tablet_manager.cc:1398] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:58:50.098886 3845 consensus_queue.cc:237] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } }
I20240503 13:58:50.100159 2787 catalog_manager.cc:5484] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 reported cstate change: term changed from 0 to 1, leader changed from <none> to 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "694cd43f32a746e7ab0e91db5785ba52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } health_report { overall_health: HEALTHY } } }
W20240503 13:58:50.205375 3921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:50.205615 3921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:50.208639 3921 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 55.4442 ms
time per row: 0.0277221 ms
I20240503 13:58:50.302024 2785 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44468:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:58:50.307030 2913 tablet_service.cc:1467] Processing CreateTablet for tablet c5d7c00e377848949299b739c6ff8d3d (DEFAULT_TABLE table=rebalance_test_table_4 [id=3860ccce8f8f4a17b1dc250f91397df8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:50.307085 2912 tablet_service.cc:1467] Processing CreateTablet for tablet e5e343562d6043b79af234ed9cdd69d4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=3860ccce8f8f4a17b1dc250f91397df8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:50.307416 2913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5d7c00e377848949299b739c6ff8d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:50.307030 2911 tablet_service.cc:1467] Processing CreateTablet for tablet 1e949fd674614e29a933cdf304d0ee94 (DEFAULT_TABLE table=rebalance_test_table_4 [id=3860ccce8f8f4a17b1dc250f91397df8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:50.307586 2912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5e343562d6043b79af234ed9cdd69d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:50.307825 2911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e949fd674614e29a933cdf304d0ee94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:50.309379 3799 tablet_bootstrap.cc:492] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d: Bootstrap starting.
I20240503 13:58:50.310098 3799 tablet_bootstrap.cc:654] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:50.310936 3799 tablet_bootstrap.cc:492] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d: No bootstrap required, opened a new log
I20240503 13:58:50.311002 3799 ts_tablet_manager.cc:1367] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 13:58:50.311337 3799 raft_consensus.cc:348] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:50.311416 3799 raft_consensus.cc:374] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:50.311442 3799 raft_consensus.cc:729] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9c26918b0724416878324492440c93d, State: Initialized, Role: FOLLOWER
I20240503 13:58:50.311589 3799 consensus_queue.cc:260] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:50.311678 3799 raft_consensus.cc:388] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:58:50.311730 3799 raft_consensus.cc:482] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:58:50.311780 3799 raft_consensus.cc:3037] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:50.312423 3799 raft_consensus.cc:504] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:50.312528 3799 leader_election.cc:302] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9c26918b0724416878324492440c93d; no voters:
I20240503 13:58:50.312598 3799 leader_election.cc:288] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:58:50.312665 3803 raft_consensus.cc:2781] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:50.312776 3803 raft_consensus.cc:686] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Becoming Leader. State: Replica: a9c26918b0724416878324492440c93d, State: Running, Role: LEADER
I20240503 13:58:50.312891 3799 ts_tablet_manager.cc:1398] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:50.312892 3803 consensus_queue.cc:237] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:50.313057 3799 tablet_bootstrap.cc:492] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d: Bootstrap starting.
I20240503 13:58:50.313848 3799 tablet_bootstrap.cc:654] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:50.314347 2785 catalog_manager.cc:5484] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d reported cstate change: term changed from 0 to 1, leader changed from <none> to a9c26918b0724416878324492440c93d (127.0.99.129). New cstate: current_term: 1 leader_uuid: "a9c26918b0724416878324492440c93d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } health_report { overall_health: HEALTHY } } }
I20240503 13:58:50.315181 3799 tablet_bootstrap.cc:492] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d: No bootstrap required, opened a new log
I20240503 13:58:50.315255 3799 ts_tablet_manager.cc:1367] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:50.315660 3799 raft_consensus.cc:348] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:50.315754 3799 raft_consensus.cc:374] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:50.315797 3799 raft_consensus.cc:729] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9c26918b0724416878324492440c93d, State: Initialized, Role: FOLLOWER
I20240503 13:58:50.315919 3799 consensus_queue.cc:260] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:50.315994 3799 raft_consensus.cc:388] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:58:50.316037 3799 raft_consensus.cc:482] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:58:50.316082 3799 raft_consensus.cc:3037] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:50.316681 3799 raft_consensus.cc:504] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:50.316797 3799 leader_election.cc:302] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9c26918b0724416878324492440c93d; no voters:
I20240503 13:58:50.316884 3799 leader_election.cc:288] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:58:50.316936 3801 raft_consensus.cc:2781] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:50.317031 3801 raft_consensus.cc:686] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [term 1 LEADER]: Becoming Leader. State: Replica: a9c26918b0724416878324492440c93d, State: Running, Role: LEADER
I20240503 13:58:50.317102 3799 ts_tablet_manager.cc:1398] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:50.317137 3801 consensus_queue.cc:237] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:50.317232 3799 tablet_bootstrap.cc:492] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d: Bootstrap starting.
I20240503 13:58:50.318044 3799 tablet_bootstrap.cc:654] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:50.318249 2785 catalog_manager.cc:5484] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d reported cstate change: term changed from 0 to 1, leader changed from <none> to a9c26918b0724416878324492440c93d (127.0.99.129). New cstate: current_term: 1 leader_uuid: "a9c26918b0724416878324492440c93d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } health_report { overall_health: HEALTHY } } }
I20240503 13:58:50.319216 3799 tablet_bootstrap.cc:492] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d: No bootstrap required, opened a new log
I20240503 13:58:50.319414 3799 ts_tablet_manager.cc:1367] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:50.319765 3799 raft_consensus.cc:348] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:50.319834 3799 raft_consensus.cc:374] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:50.319869 3799 raft_consensus.cc:729] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9c26918b0724416878324492440c93d, State: Initialized, Role: FOLLOWER
I20240503 13:58:50.319967 3799 consensus_queue.cc:260] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:50.320041 3799 raft_consensus.cc:388] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:58:50.320079 3799 raft_consensus.cc:482] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:58:50.320120 3799 raft_consensus.cc:3037] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:50.320684 3799 raft_consensus.cc:504] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:50.320781 3799 leader_election.cc:302] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9c26918b0724416878324492440c93d; no voters:
I20240503 13:58:50.320869 3799 leader_election.cc:288] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:58:50.320909 3803 raft_consensus.cc:2781] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:50.320974 3803 raft_consensus.cc:686] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Becoming Leader. State: Replica: a9c26918b0724416878324492440c93d, State: Running, Role: LEADER
I20240503 13:58:50.321110 3799 ts_tablet_manager.cc:1398] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 13:58:50.321072 3803 consensus_queue.cc:237] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } }
I20240503 13:58:50.322249 2785 catalog_manager.cc:5484] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d reported cstate change: term changed from 0 to 1, leader changed from <none> to a9c26918b0724416878324492440c93d (127.0.99.129). New cstate: current_term: 1 leader_uuid: "a9c26918b0724416878324492440c93d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } health_report { overall_health: HEALTHY } } }
W20240503 13:58:50.428400 3951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:50.428596 3951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:50.431270 3951 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 55.3227 ms
time per row: 0.0276613 ms
I20240503 13:58:50.525353 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:44145
--local_ip_for_outbound_sockets=127.0.99.131
--tserver_master_addrs=127.0.99.190:37103
--webserver_port=34273
--webserver_interface=127.0.99.131
--builtin_ntp_servers=127.0.99.148:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 13:58:50.628648 3975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:50.628847 3975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:50.628912 3975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:50.631796 3975 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:50.631860 3975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:50.631943 3975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 13:58:50.635438 3975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:44145
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=34273
--tserver_master_addrs=127.0.99.190:37103
--never_fsync=true
--heap_profile_path=/tmp/kudu.3975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:50.636237 3975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:50.637339 3975 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:50.645573 3981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:50.646363 3980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:50.647181 3983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:50.647956 3975 server_base.cc:1034] running on GCE node
I20240503 13:58:50.648360 3975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:50.648876 3975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:50.650033 3975 hybrid_clock.cc:648] HybridClock initialized: now 1714744730650025 us; error 24 us; skew 500 ppm
I20240503 13:58:50.651834 3975 webserver.cc:460] Webserver started at http://127.0.99.131:34273/ using document root <none> and password file <none>
I20240503 13:58:50.652364 3975 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:50.652439 3975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:50.655572 3975 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:58:50.657310 3989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:50.658215 3975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:50.658315 3975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9b3cf5210e6b4f2188e66a80dca823e6"
format_stamp: "Formatted at 2024-05-03 13:58:44 on dist-test-slave-fr9h"
I20240503 13:58:50.658690 3975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:50.687036 3975 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:50.687768 3975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:50.687939 3975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:50.688477 3975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:50.689349 3975 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:50.689399 3975 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:50.689457 3975 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:50.689484 3975 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:50.697957 3975 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:44145
I20240503 13:58:50.697994 4102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:44145 every 8 connection(s)
I20240503 13:58:50.698879 3975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 13:58:50.700487 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 3975
I20240503 13:58:50.700927 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:40887
--local_ip_for_outbound_sockets=127.0.99.132
--tserver_master_addrs=127.0.99.190:37103
--webserver_port=42019
--webserver_interface=127.0.99.132
--builtin_ntp_servers=127.0.99.148:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:58:50.708670 4103 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37103
I20240503 13:58:50.708915 4103 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:50.709605 4103 heartbeater.cc:511] Master 127.0.99.190:37103 requested a full tablet report, sending...
I20240503 13:58:50.710772 2790 ts_manager.cc:194] Re-registered known tserver with Master: 9b3cf5210e6b4f2188e66a80dca823e6 (127.0.99.131:44145)
I20240503 13:58:50.711354 2790 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:50195
W20240503 13:58:50.806594 4106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:50.806790 4106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:50.806849 4106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:50.809794 4106 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:50.809854 4106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:50.809960 4106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 13:58:50.813411 4106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:40887
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=42019
--tserver_master_addrs=127.0.99.190:37103
--never_fsync=true
--heap_profile_path=/tmp/kudu.4106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:50.814209 4106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:50.815339 4106 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:50.824612 4114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:50.825387 4112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:50.826203 4111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:50.827132 4106 server_base.cc:1034] running on GCE node
I20240503 13:58:50.827517 4106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:50.828042 4106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:50.829214 4106 hybrid_clock.cc:648] HybridClock initialized: now 1714744730829204 us; error 22 us; skew 500 ppm
I20240503 13:58:50.830989 4106 webserver.cc:460] Webserver started at http://127.0.99.132:42019/ using document root <none> and password file <none>
I20240503 13:58:50.831568 4106 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:50.831656 4106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:50.835188 4106 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20240503 13:58:50.836923 4120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:50.837802 4106 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:50.837896 4106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e488cf84f7b74419a218d3512e237fea"
format_stamp: "Formatted at 2024-05-03 13:58:44 on dist-test-slave-fr9h"
I20240503 13:58:50.838263 4106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:50.854041 4106 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:50.854593 4106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:50.854733 4106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:50.855206 4106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:50.856003 4106 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:50.856062 4106 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:50.856099 4106 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:50.856118 4106 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:50.864077 4106 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:40887
I20240503 13:58:50.864105 4233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:40887 every 8 connection(s)
I20240503 13:58:50.864925 4106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 13:58:50.866359 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 4106
I20240503 13:58:50.866750 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:45407
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:37103
--webserver_port=45247
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:58:50.873102 4234 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37103
I20240503 13:58:50.873296 4234 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:50.873719 4234 heartbeater.cc:511] Master 127.0.99.190:37103 requested a full tablet report, sending...
I20240503 13:58:50.874619 2790 ts_manager.cc:194] Re-registered known tserver with Master: e488cf84f7b74419a218d3512e237fea (127.0.99.132:40887)
I20240503 13:58:50.875088 2790 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:57837
W20240503 13:58:50.965935 4237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:50.966117 4237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:50.966187 4237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:50.968943 4237 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:50.969008 4237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:50.969094 4237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 13:58:50.972561 4237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:45407
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=45247
--tserver_master_addrs=127.0.99.190:37103
--never_fsync=true
--heap_profile_path=/tmp/kudu.4237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:50.973316 4237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:50.974339 4237 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:50.982484 4245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:50.983238 4243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:50.984191 4237 server_base.cc:1034] running on GCE node
W20240503 13:58:50.984728 4242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:50.985049 4237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:50.985494 4237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:50.986649 4237 hybrid_clock.cc:648] HybridClock initialized: now 1714744730986639 us; error 22 us; skew 500 ppm
I20240503 13:58:50.988283 4237 webserver.cc:460] Webserver started at http://127.0.99.133:45247/ using document root <none> and password file <none>
I20240503 13:58:50.988747 4237 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:50.988806 4237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:50.991899 4237 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:58:50.993613 4251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:50.994555 4237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:50.994650 4237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "acc5e87f231a46f5beb89cc24cbe28c4"
format_stamp: "Formatted at 2024-05-03 13:58:45 on dist-test-slave-fr9h"
I20240503 13:58:50.995067 4237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:51.030496 4237 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:51.031098 4237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:51.031250 4237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:51.031798 4237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:51.032629 4237 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:51.032678 4237 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:51.032737 4237 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:51.032763 4237 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:51.040596 4237 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:45407
I20240503 13:58:51.040622 4364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:45407 every 8 connection(s)
I20240503 13:58:51.041586 4237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 13:58:51.041852 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 4237
I20240503 13:58:51.042272 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:38729
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:37103
--webserver_port=46313
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:58:51.050907 4365 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37103
I20240503 13:58:51.051131 4365 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:51.051774 4365 heartbeater.cc:511] Master 127.0.99.190:37103 requested a full tablet report, sending...
I20240503 13:58:51.052886 2790 ts_manager.cc:194] Re-registered known tserver with Master: acc5e87f231a46f5beb89cc24cbe28c4 (127.0.99.133:45407)
I20240503 13:58:51.053598 2790 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:43281
W20240503 13:58:51.144924 4367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:51.145105 4367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:51.145143 4367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:51.147764 4367 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:51.147814 4367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:51.147869 4367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 13:58:51.150910 4367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:38729
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=46313
--tserver_master_addrs=127.0.99.190:37103
--never_fsync=true
--heap_profile_path=/tmp/kudu.4367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:51.151674 4367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:51.152679 4367 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:51.160560 4376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:51.161414 4374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:51.162206 4373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:51.162976 4367 server_base.cc:1034] running on GCE node
I20240503 13:58:51.163306 4367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:51.163794 4367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:51.164952 4367 hybrid_clock.cc:648] HybridClock initialized: now 1714744731164940 us; error 21 us; skew 500 ppm
I20240503 13:58:51.166450 4367 webserver.cc:460] Webserver started at http://127.0.99.134:46313/ using document root <none> and password file <none>
I20240503 13:58:51.166909 4367 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:51.166971 4367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:51.170001 4367 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:51.171617 4382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:51.172556 4367 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:51.172653 4367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "3b1cfcd5138346b6ac5b8581c536034d"
format_stamp: "Formatted at 2024-05-03 13:58:45 on dist-test-slave-fr9h"
I20240503 13:58:51.173046 4367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:51.205147 4367 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:51.205775 4367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:51.205930 4367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:51.206509 4367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:51.207408 4367 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:51.207456 4367 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:51.207509 4367 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:51.207540 4367 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:51.216898 4367 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:38729
I20240503 13:58:51.216919 4495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:38729 every 8 connection(s)
I20240503 13:58:51.217847 4367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 13:58:51.219430 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 4367
I20240503 13:58:51.219825 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:33161
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:37103
--webserver_port=46583
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:58:51.227200 4496 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37103
I20240503 13:58:51.227486 4496 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:51.228163 4496 heartbeater.cc:511] Master 127.0.99.190:37103 requested a full tablet report, sending...
I20240503 13:58:51.229235 2790 ts_manager.cc:194] Re-registered known tserver with Master: 3b1cfcd5138346b6ac5b8581c536034d (127.0.99.134:38729)
I20240503 13:58:51.229758 2790 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:50935
W20240503 13:58:51.327492 4499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:51.327680 4499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:51.327739 4499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:51.330528 4499 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:51.330581 4499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:51.330641 4499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 13:58:51.334134 4499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:33161
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=46583
--tserver_master_addrs=127.0.99.190:37103
--never_fsync=true
--heap_profile_path=/tmp/kudu.4499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:51.335063 4499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:51.336196 4499 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:51.344666 4504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:51.345441 4507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:51.346347 4505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:51.347210 4499 server_base.cc:1034] running on GCE node
I20240503 13:58:51.347561 4499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:51.348062 4499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:51.349208 4499 hybrid_clock.cc:648] HybridClock initialized: now 1714744731349188 us; error 30 us; skew 500 ppm
I20240503 13:58:51.350955 4499 webserver.cc:460] Webserver started at http://127.0.99.135:46583/ using document root <none> and password file <none>
I20240503 13:58:51.351511 4499 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:51.351588 4499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:51.354679 4499 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:58:51.356417 4513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:51.357353 4499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:51.357450 4499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "5c33992e399d4e63a2cc07cb4a479316"
format_stamp: "Formatted at 2024-05-03 13:58:45 on dist-test-slave-fr9h"
I20240503 13:58:51.357841 4499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:51.376399 4499 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:51.377030 4499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:51.377177 4499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:51.377702 4499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:51.378543 4499 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:51.378590 4499 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:51.378664 4499 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:51.378696 4499 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:51.387744 4499 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:33161
I20240503 13:58:51.387763 4626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:33161 every 8 connection(s)
I20240503 13:58:51.388686 4499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 13:58:51.395203 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 4499
I20240503 13:58:51.397086 4627 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37103
I20240503 13:58:51.397284 4627 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:51.397764 4627 heartbeater.cc:511] Master 127.0.99.190:37103 requested a full tablet report, sending...
I20240503 13:58:51.398738 2790 ts_manager.cc:194] Re-registered known tserver with Master: 5c33992e399d4e63a2cc07cb4a479316 (127.0.99.135:33161)
I20240503 13:58:51.399369 2790 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:59905
I20240503 13:58:51.583814 2912 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:58:51.584743 3042 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:58:51.586707 4561 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:51.588047 4430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:51.591562 4299 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:51.603869 4037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:51.616600 4168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:51.709539 4430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:51.711585 4561 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:51.711879 3042 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:58:51.712406 4037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:51.713963 4103 heartbeater.cc:503] Master 127.0.99.190:37103 was elected leader, sending a full tablet report...
I20240503 13:58:51.714859 4299 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:51.730011 2912 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:58:51.732259 4168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:51.763906 3064 consensus_queue.cc:237] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: true } }
I20240503 13:58:51.765794 4718 raft_consensus.cc:2932] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER acc5e87f231a46f5beb89cc24cbe28c4 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: true } } }
I20240503 13:58:51.768841 2782 catalog_manager.cc:5484] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 reported cstate change: config changed from index -1 to 4, NON_VOTER acc5e87f231a46f5beb89cc24cbe28c4 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "694cd43f32a746e7ab0e91db5785ba52" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 13:58:51.771119 2997 consensus_peers.cc:485] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 -> Peer acc5e87f231a46f5beb89cc24cbe28c4 (127.0.99.133:45407): Couldn't send request to peer acc5e87f231a46f5beb89cc24cbe28c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fa9ffe2aeedf42e89d4e00ac12fb9e92. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:51.776234 2929 consensus_queue.cc:237] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: true } }
I20240503 13:58:51.777546 4731 raft_consensus.cc:2932] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b1cfcd5138346b6ac5b8581c536034d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: true } } }
I20240503 13:58:51.780045 2782 catalog_manager.cc:5484] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d reported cstate change: config changed from index -1 to 4, NON_VOTER 3b1cfcd5138346b6ac5b8581c536034d (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "a9c26918b0724416878324492440c93d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 13:58:51.782297 2868 consensus_peers.cc:485] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d -> Peer 3b1cfcd5138346b6ac5b8581c536034d (127.0.99.134:38729): Couldn't send request to peer 3b1cfcd5138346b6ac5b8581c536034d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 558e844e3d344fcbb0e41cdf132c880c. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:51.787159 2929 consensus_queue.cc:237] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: true } }
I20240503 13:58:51.788012 4730 raft_consensus.cc:2932] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5c33992e399d4e63a2cc07cb4a479316 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: true } } }
I20240503 13:58:51.790143 2782 catalog_manager.cc:5484] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d reported cstate change: config changed from index -1 to 4, NON_VOTER 5c33992e399d4e63a2cc07cb4a479316 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "a9c26918b0724416878324492440c93d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 13:58:51.792320 2868 consensus_peers.cc:485] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d -> Peer 5c33992e399d4e63a2cc07cb4a479316 (127.0.99.135:33161): Couldn't send request to peer 5c33992e399d4e63a2cc07cb4a479316. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1e949fd674614e29a933cdf304d0ee94. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:51.797528 3064 consensus_queue.cc:237] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: true } }
I20240503 13:58:51.798501 4718 raft_consensus.cc:2932] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e488cf84f7b74419a218d3512e237fea (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: true } } }
I20240503 13:58:51.800737 2782 catalog_manager.cc:5484] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 reported cstate change: config changed from index -1 to 4, NON_VOTER e488cf84f7b74419a218d3512e237fea (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "694cd43f32a746e7ab0e91db5785ba52" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 13:58:51.802949 3000 consensus_peers.cc:485] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 -> Peer e488cf84f7b74419a218d3512e237fea (127.0.99.132:40887): Couldn't send request to peer e488cf84f7b74419a218d3512e237fea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f599173c7624329af379768658f6041. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:51.807356 2929 consensus_queue.cc:237] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: true } }
I20240503 13:58:51.808372 4731 raft_consensus.cc:2932] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9b3cf5210e6b4f2188e66a80dca823e6 (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: true } } }
I20240503 13:58:51.810804 2782 catalog_manager.cc:5484] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d reported cstate change: config changed from index -1 to 4, NON_VOTER 9b3cf5210e6b4f2188e66a80dca823e6 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "a9c26918b0724416878324492440c93d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 13:58:51.812712 2866 consensus_peers.cc:485] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d -> Peer 9b3cf5210e6b4f2188e66a80dca823e6 (127.0.99.131:44145): Couldn't send request to peer 9b3cf5210e6b4f2188e66a80dca823e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce166608b56540d89bddc4919356ebe7. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:51.817967 3064 consensus_queue.cc:237] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: true } }
I20240503 13:58:51.818866 4717 raft_consensus.cc:2932] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER acc5e87f231a46f5beb89cc24cbe28c4 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: true } } }
W20240503 13:58:51.819272 2997 consensus_peers.cc:485] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 -> Peer acc5e87f231a46f5beb89cc24cbe28c4 (127.0.99.133:45407): Couldn't send request to peer acc5e87f231a46f5beb89cc24cbe28c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a88836924b04bfea8524184d061a19b. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:51.821213 2782 catalog_manager.cc:5484] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 reported cstate change: config changed from index -1 to 4, NON_VOTER acc5e87f231a46f5beb89cc24cbe28c4 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "694cd43f32a746e7ab0e91db5785ba52" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:51.828532 2929 consensus_queue.cc:237] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: true } }
I20240503 13:58:51.829363 4730 raft_consensus.cc:2932] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b1cfcd5138346b6ac5b8581c536034d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: true } } }
W20240503 13:58:51.830004 2868 consensus_peers.cc:485] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d -> Peer 3b1cfcd5138346b6ac5b8581c536034d (127.0.99.134:38729): Couldn't send request to peer 3b1cfcd5138346b6ac5b8581c536034d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03e82ebc742b46ce98fc9ab599b08f46. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:51.831549 2782 catalog_manager.cc:5484] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d reported cstate change: config changed from index -1 to 4, NON_VOTER 3b1cfcd5138346b6ac5b8581c536034d (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "a9c26918b0724416878324492440c93d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:51.837729 2929 consensus_queue.cc:237] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: true } }
I20240503 13:58:51.838456 4763 raft_consensus.cc:2932] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5c33992e399d4e63a2cc07cb4a479316 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: true } } }
W20240503 13:58:51.838924 2868 consensus_peers.cc:485] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d -> Peer 5c33992e399d4e63a2cc07cb4a479316 (127.0.99.135:33161): Couldn't send request to peer 5c33992e399d4e63a2cc07cb4a479316. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25a1559451b8472482ac6e482da41d28. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:51.840651 2782 catalog_manager.cc:5484] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d reported cstate change: config changed from index -1 to 4, NON_VOTER 5c33992e399d4e63a2cc07cb4a479316 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "a9c26918b0724416878324492440c93d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:51.846612 3064 consensus_queue.cc:237] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: true } }
I20240503 13:58:51.847486 4718 raft_consensus.cc:2932] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e488cf84f7b74419a218d3512e237fea (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: true } } }
W20240503 13:58:51.847930 3000 consensus_peers.cc:485] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 -> Peer e488cf84f7b74419a218d3512e237fea (127.0.99.132:40887): Couldn't send request to peer e488cf84f7b74419a218d3512e237fea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d899ba5a20f0466da4e0e3fc9831e5f5. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:51.849875 2789 catalog_manager.cc:5484] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 reported cstate change: config changed from index -1 to 4, NON_VOTER e488cf84f7b74419a218d3512e237fea (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "694cd43f32a746e7ab0e91db5785ba52" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:51.876827 4234 heartbeater.cc:503] Master 127.0.99.190:37103 was elected leader, sending a full tablet report...
I20240503 13:58:52.055742 4365 heartbeater.cc:503] Master 127.0.99.190:37103 was elected leader, sending a full tablet report...
I20240503 13:58:52.232062 4496 heartbeater.cc:503] Master 127.0.99.190:37103 was elected leader, sending a full tablet report...
I20240503 13:58:52.261557 4866 ts_tablet_manager.cc:902] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d: Initiating tablet copy from peer a9c26918b0724416878324492440c93d (127.0.99.129:39461)
I20240503 13:58:52.262339 4866 tablet_copy_client.cc:308] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:39461
I20240503 13:58:52.265849 2953 tablet_copy_service.cc:140] P a9c26918b0724416878324492440c93d: Received BeginTabletCopySession request for tablet 558e844e3d344fcbb0e41cdf132c880c from peer 3b1cfcd5138346b6ac5b8581c536034d ({username='slave'} at 127.0.99.134:46203)
I20240503 13:58:52.265975 2953 tablet_copy_service.cc:161] P a9c26918b0724416878324492440c93d: Beginning new tablet copy session on tablet 558e844e3d344fcbb0e41cdf132c880c from peer 3b1cfcd5138346b6ac5b8581c536034d at {username='slave'} at 127.0.99.134:46203: session id = 3b1cfcd5138346b6ac5b8581c536034d-558e844e3d344fcbb0e41cdf132c880c
I20240503 13:58:52.267115 2953 tablet_copy_source_session.cc:205] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:58:52.269996 4866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 558e844e3d344fcbb0e41cdf132c880c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:52.272758 4872 ts_tablet_manager.cc:902] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d: Initiating tablet copy from peer a9c26918b0724416878324492440c93d (127.0.99.129:39461)
I20240503 13:58:52.273284 4872 tablet_copy_client.cc:308] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:39461
I20240503 13:58:52.273746 2953 tablet_copy_service.cc:140] P a9c26918b0724416878324492440c93d: Received BeginTabletCopySession request for tablet 03e82ebc742b46ce98fc9ab599b08f46 from peer 3b1cfcd5138346b6ac5b8581c536034d ({username='slave'} at 127.0.99.134:46203)
I20240503 13:58:52.273838 2953 tablet_copy_service.cc:161] P a9c26918b0724416878324492440c93d: Beginning new tablet copy session on tablet 03e82ebc742b46ce98fc9ab599b08f46 from peer 3b1cfcd5138346b6ac5b8581c536034d at {username='slave'} at 127.0.99.134:46203: session id = 3b1cfcd5138346b6ac5b8581c536034d-03e82ebc742b46ce98fc9ab599b08f46
I20240503 13:58:52.274607 2953 tablet_copy_source_session.cc:205] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:58:52.275264 4872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e82ebc742b46ce98fc9ab599b08f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:52.275460 4866 tablet_copy_client.cc:783] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d: tablet copy: Starting download of 0 data blocks...
I20240503 13:58:52.275652 4866 tablet_copy_client.cc:647] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:58:52.276914 4872 tablet_copy_client.cc:783] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d: tablet copy: Starting download of 0 data blocks...
I20240503 13:58:52.277058 4872 tablet_copy_client.cc:647] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:58:52.277119 4866 tablet_copy_client.cc:523] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:58:52.278080 4872 tablet_copy_client.cc:523] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:58:52.278704 4866 tablet_bootstrap.cc:492] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d: Bootstrap starting.
I20240503 13:58:52.279095 4872 tablet_bootstrap.cc:492] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d: Bootstrap starting.
I20240503 13:58:52.285138 4872 log.cc:824] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:52.285140 4866 log.cc:824] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:52.288050 4879 ts_tablet_manager.cc:902] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316: Initiating tablet copy from peer a9c26918b0724416878324492440c93d (127.0.99.129:39461)
I20240503 13:58:52.288741 4879 tablet_copy_client.cc:308] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:39461
I20240503 13:58:52.292155 2953 tablet_copy_service.cc:140] P a9c26918b0724416878324492440c93d: Received BeginTabletCopySession request for tablet 25a1559451b8472482ac6e482da41d28 from peer 5c33992e399d4e63a2cc07cb4a479316 ({username='slave'} at 127.0.99.135:51983)
I20240503 13:58:52.292244 2953 tablet_copy_service.cc:161] P a9c26918b0724416878324492440c93d: Beginning new tablet copy session on tablet 25a1559451b8472482ac6e482da41d28 from peer 5c33992e399d4e63a2cc07cb4a479316 at {username='slave'} at 127.0.99.135:51983: session id = 5c33992e399d4e63a2cc07cb4a479316-25a1559451b8472482ac6e482da41d28
I20240503 13:58:52.292863 2953 tablet_copy_source_session.cc:205] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:58:52.295543 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25a1559451b8472482ac6e482da41d28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:52.300246 4879 tablet_copy_client.cc:783] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316: tablet copy: Starting download of 0 data blocks...
I20240503 13:58:52.300518 4879 tablet_copy_client.cc:647] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316: tablet copy: Starting download of 1 WAL segments...
I20240503 13:58:52.301774 4879 tablet_copy_client.cc:523] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:58:52.303258 4879 tablet_bootstrap.cc:492] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316: Bootstrap starting.
I20240503 13:58:52.306228 4885 ts_tablet_manager.cc:902] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4: Initiating tablet copy from peer 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130:34403)
I20240503 13:58:52.307041 4885 tablet_copy_client.cc:308] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34403
I20240503 13:58:52.309188 4879 log.cc:824] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:52.311317 4866 tablet_bootstrap.cc:492] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:58:52.311661 4866 tablet_bootstrap.cc:492] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d: Bootstrap complete.
I20240503 13:58:52.311746 4872 tablet_bootstrap.cc:492] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:58:52.311980 4872 tablet_bootstrap.cc:492] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d: Bootstrap complete.
I20240503 13:58:52.311990 4866 ts_tablet_manager.cc:1367] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d: Time spent bootstrapping tablet: real 0.033s user 0.033s sys 0.000s
I20240503 13:58:52.312141 4872 ts_tablet_manager.cc:1367] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d: Time spent bootstrapping tablet: real 0.033s user 0.027s sys 0.005s
I20240503 13:58:52.313261 3084 tablet_copy_service.cc:140] P 694cd43f32a746e7ab0e91db5785ba52: Received BeginTabletCopySession request for tablet 1a88836924b04bfea8524184d061a19b from peer acc5e87f231a46f5beb89cc24cbe28c4 ({username='slave'} at 127.0.99.133:54557)
I20240503 13:58:52.313423 3084 tablet_copy_service.cc:161] P 694cd43f32a746e7ab0e91db5785ba52: Beginning new tablet copy session on tablet 1a88836924b04bfea8524184d061a19b from peer acc5e87f231a46f5beb89cc24cbe28c4 at {username='slave'} at 127.0.99.133:54557: session id = acc5e87f231a46f5beb89cc24cbe28c4-1a88836924b04bfea8524184d061a19b
I20240503 13:58:52.314349 3084 tablet_copy_source_session.cc:205] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:58:52.316231 4866 raft_consensus.cc:348] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: true } }
I20240503 13:58:52.316511 4866 raft_consensus.cc:729] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b1cfcd5138346b6ac5b8581c536034d, State: Initialized, Role: LEARNER
I20240503 13:58:52.316231 4872 raft_consensus.cc:348] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: true } }
I20240503 13:58:52.316654 4872 raft_consensus.cc:729] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b1cfcd5138346b6ac5b8581c536034d, State: Initialized, Role: LEARNER
I20240503 13:58:52.316841 4866 consensus_queue.cc:260] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: true } }
I20240503 13:58:52.316841 4872 consensus_queue.cc:260] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: true } }
I20240503 13:58:52.317575 4885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a88836924b04bfea8524184d061a19b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:52.317621 4866 ts_tablet_manager.cc:1398] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 13:58:52.317668 4872 ts_tablet_manager.cc:1398] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 13:58:52.319204 2953 tablet_copy_service.cc:342] P a9c26918b0724416878324492440c93d: Request end of tablet copy session 3b1cfcd5138346b6ac5b8581c536034d-03e82ebc742b46ce98fc9ab599b08f46 received from {username='slave'} at 127.0.99.134:46203
I20240503 13:58:52.319324 2953 tablet_copy_service.cc:433] P a9c26918b0724416878324492440c93d: ending tablet copy session 3b1cfcd5138346b6ac5b8581c536034d-03e82ebc742b46ce98fc9ab599b08f46 on tablet 03e82ebc742b46ce98fc9ab599b08f46 with peer 3b1cfcd5138346b6ac5b8581c536034d
I20240503 13:58:52.319720 2952 tablet_copy_service.cc:342] P a9c26918b0724416878324492440c93d: Request end of tablet copy session 3b1cfcd5138346b6ac5b8581c536034d-558e844e3d344fcbb0e41cdf132c880c received from {username='slave'} at 127.0.99.134:46203
I20240503 13:58:52.319774 2952 tablet_copy_service.cc:433] P a9c26918b0724416878324492440c93d: ending tablet copy session 3b1cfcd5138346b6ac5b8581c536034d-558e844e3d344fcbb0e41cdf132c880c on tablet 558e844e3d344fcbb0e41cdf132c880c with peer 3b1cfcd5138346b6ac5b8581c536034d
I20240503 13:58:52.323240 4895 ts_tablet_manager.cc:902] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6: Initiating tablet copy from peer a9c26918b0724416878324492440c93d (127.0.99.129:39461)
I20240503 13:58:52.323802 4892 ts_tablet_manager.cc:902] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea: Initiating tablet copy from peer 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130:34403)
I20240503 13:58:52.324069 4895 tablet_copy_client.cc:308] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:39461
I20240503 13:58:52.324482 4892 tablet_copy_client.cc:308] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34403
I20240503 13:58:52.325533 4885 tablet_copy_client.cc:783] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4: tablet copy: Starting download of 0 data blocks...
I20240503 13:58:52.325701 4885 tablet_copy_client.cc:647] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4: tablet copy: Starting download of 1 WAL segments...
I20240503 13:58:52.327245 4885 tablet_copy_client.cc:523] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:58:52.329071 3084 tablet_copy_service.cc:140] P 694cd43f32a746e7ab0e91db5785ba52: Received BeginTabletCopySession request for tablet d899ba5a20f0466da4e0e3fc9831e5f5 from peer e488cf84f7b74419a218d3512e237fea ({username='slave'} at 127.0.99.132:49745)
I20240503 13:58:52.329167 3084 tablet_copy_service.cc:161] P 694cd43f32a746e7ab0e91db5785ba52: Beginning new tablet copy session on tablet d899ba5a20f0466da4e0e3fc9831e5f5 from peer e488cf84f7b74419a218d3512e237fea at {username='slave'} at 127.0.99.132:49745: session id = e488cf84f7b74419a218d3512e237fea-d899ba5a20f0466da4e0e3fc9831e5f5
I20240503 13:58:52.329756 2952 tablet_copy_service.cc:140] P a9c26918b0724416878324492440c93d: Received BeginTabletCopySession request for tablet ce166608b56540d89bddc4919356ebe7 from peer 9b3cf5210e6b4f2188e66a80dca823e6 ({username='slave'} at 127.0.99.131:38483)
I20240503 13:58:52.329850 2952 tablet_copy_service.cc:161] P a9c26918b0724416878324492440c93d: Beginning new tablet copy session on tablet ce166608b56540d89bddc4919356ebe7 from peer 9b3cf5210e6b4f2188e66a80dca823e6 at {username='slave'} at 127.0.99.131:38483: session id = 9b3cf5210e6b4f2188e66a80dca823e6-ce166608b56540d89bddc4919356ebe7
I20240503 13:58:52.329972 3084 tablet_copy_source_session.cc:205] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:58:52.330338 4885 tablet_bootstrap.cc:492] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4: Bootstrap starting.
I20240503 13:58:52.330667 2952 tablet_copy_source_session.cc:205] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:58:52.332832 4892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d899ba5a20f0466da4e0e3fc9831e5f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:52.333732 4895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce166608b56540d89bddc4919356ebe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:52.336133 4885 log.cc:824] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:52.339161 4892 tablet_copy_client.cc:783] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea: tablet copy: Starting download of 0 data blocks...
I20240503 13:58:52.339366 4892 tablet_copy_client.cc:647] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea: tablet copy: Starting download of 1 WAL segments...
I20240503 13:58:52.339462 4895 tablet_copy_client.cc:783] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6: tablet copy: Starting download of 0 data blocks...
I20240503 13:58:52.339649 4895 tablet_copy_client.cc:647] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6: tablet copy: Starting download of 1 WAL segments...
I20240503 13:58:52.340364 4906 ts_tablet_manager.cc:902] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4: Initiating tablet copy from peer 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130:34403)
I20240503 13:58:52.340893 4895 tablet_copy_client.cc:523] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:58:52.340898 4892 tablet_copy_client.cc:523] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:58:52.341020 4879 tablet_bootstrap.cc:492] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:58:52.341372 4879 tablet_bootstrap.cc:492] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316: Bootstrap complete.
I20240503 13:58:52.341653 4879 ts_tablet_manager.cc:1367] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316: Time spent bootstrapping tablet: real 0.039s user 0.026s sys 0.008s
I20240503 13:58:52.342630 4892 tablet_bootstrap.cc:492] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea: Bootstrap starting.
I20240503 13:58:52.342630 4895 tablet_bootstrap.cc:492] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6: Bootstrap starting.
I20240503 13:58:52.344029 4879 raft_consensus.cc:348] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: true } }
I20240503 13:58:52.344252 4879 raft_consensus.cc:729] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5c33992e399d4e63a2cc07cb4a479316, State: Initialized, Role: LEARNER
I20240503 13:58:52.344594 4879 consensus_queue.cc:260] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: true } }
I20240503 13:58:52.345419 4879 ts_tablet_manager.cc:1398] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 13:58:52.345758 4627 heartbeater.cc:503] Master 127.0.99.190:37103 was elected leader, sending a full tablet report...
I20240503 13:58:52.346798 2952 tablet_copy_service.cc:342] P a9c26918b0724416878324492440c93d: Request end of tablet copy session 5c33992e399d4e63a2cc07cb4a479316-25a1559451b8472482ac6e482da41d28 received from {username='slave'} at 127.0.99.135:51983
I20240503 13:58:52.346896 2952 tablet_copy_service.cc:433] P a9c26918b0724416878324492440c93d: ending tablet copy session 5c33992e399d4e63a2cc07cb4a479316-25a1559451b8472482ac6e482da41d28 on tablet 25a1559451b8472482ac6e482da41d28 with peer 5c33992e399d4e63a2cc07cb4a479316
I20240503 13:58:52.348826 4892 log.cc:824] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:52.349253 4895 log.cc:824] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:52.349481 4906 tablet_copy_client.cc:308] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34403
I20240503 13:58:52.349978 3084 tablet_copy_service.cc:140] P 694cd43f32a746e7ab0e91db5785ba52: Received BeginTabletCopySession request for tablet fa9ffe2aeedf42e89d4e00ac12fb9e92 from peer acc5e87f231a46f5beb89cc24cbe28c4 ({username='slave'} at 127.0.99.133:54557)
I20240503 13:58:52.350083 3084 tablet_copy_service.cc:161] P 694cd43f32a746e7ab0e91db5785ba52: Beginning new tablet copy session on tablet fa9ffe2aeedf42e89d4e00ac12fb9e92 from peer acc5e87f231a46f5beb89cc24cbe28c4 at {username='slave'} at 127.0.99.133:54557: session id = acc5e87f231a46f5beb89cc24cbe28c4-fa9ffe2aeedf42e89d4e00ac12fb9e92
I20240503 13:58:52.351001 3084 tablet_copy_source_session.cc:205] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:58:52.352491 4906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa9ffe2aeedf42e89d4e00ac12fb9e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:52.354728 4906 tablet_copy_client.cc:783] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4: tablet copy: Starting download of 0 data blocks...
I20240503 13:58:52.354871 4906 tablet_copy_client.cc:647] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4: tablet copy: Starting download of 1 WAL segments...
I20240503 13:58:52.358948 4879 ts_tablet_manager.cc:902] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316: Initiating tablet copy from peer a9c26918b0724416878324492440c93d (127.0.99.129:39461)
I20240503 13:58:52.359344 4879 tablet_copy_client.cc:308] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:39461
I20240503 13:58:52.359735 2952 tablet_copy_service.cc:140] P a9c26918b0724416878324492440c93d: Received BeginTabletCopySession request for tablet 1e949fd674614e29a933cdf304d0ee94 from peer 5c33992e399d4e63a2cc07cb4a479316 ({username='slave'} at 127.0.99.135:51983)
I20240503 13:58:52.359782 4906 tablet_copy_client.cc:523] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:58:52.359845 2952 tablet_copy_service.cc:161] P a9c26918b0724416878324492440c93d: Beginning new tablet copy session on tablet 1e949fd674614e29a933cdf304d0ee94 from peer 5c33992e399d4e63a2cc07cb4a479316 at {username='slave'} at 127.0.99.135:51983: session id = 5c33992e399d4e63a2cc07cb4a479316-1e949fd674614e29a933cdf304d0ee94
I20240503 13:58:52.360746 2952 tablet_copy_source_session.cc:205] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:58:52.360849 4906 tablet_bootstrap.cc:492] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4: Bootstrap starting.
I20240503 13:58:52.361591 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e949fd674614e29a933cdf304d0ee94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:52.363596 4879 tablet_copy_client.cc:783] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316: tablet copy: Starting download of 0 data blocks...
I20240503 13:58:52.363737 4879 tablet_copy_client.cc:647] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316: tablet copy: Starting download of 1 WAL segments...
I20240503 13:58:52.366453 4879 tablet_copy_client.cc:523] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:58:52.368122 4879 tablet_bootstrap.cc:492] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316: Bootstrap starting.
I20240503 13:58:52.378453 4885 tablet_bootstrap.cc:492] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:58:52.378787 4885 tablet_bootstrap.cc:492] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4: Bootstrap complete.
I20240503 13:58:52.379091 4885 ts_tablet_manager.cc:1367] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4: Time spent bootstrapping tablet: real 0.050s user 0.024s sys 0.008s
I20240503 13:58:52.381601 4885 raft_consensus.cc:348] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: true } }
I20240503 13:58:52.381853 4885 raft_consensus.cc:729] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: acc5e87f231a46f5beb89cc24cbe28c4, State: Initialized, Role: LEARNER
I20240503 13:58:52.382201 4885 consensus_queue.cc:260] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: true } }
I20240503 13:58:52.383075 4885 ts_tablet_manager.cc:1398] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240503 13:58:52.386984 3084 tablet_copy_service.cc:342] P 694cd43f32a746e7ab0e91db5785ba52: Request end of tablet copy session acc5e87f231a46f5beb89cc24cbe28c4-1a88836924b04bfea8524184d061a19b received from {username='slave'} at 127.0.99.133:54557
I20240503 13:58:52.387082 3084 tablet_copy_service.cc:433] P 694cd43f32a746e7ab0e91db5785ba52: ending tablet copy session acc5e87f231a46f5beb89cc24cbe28c4-1a88836924b04bfea8524184d061a19b on tablet 1a88836924b04bfea8524184d061a19b with peer acc5e87f231a46f5beb89cc24cbe28c4
I20240503 13:58:52.388767 4892 tablet_bootstrap.cc:492] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:58:52.389083 4892 tablet_bootstrap.cc:492] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea: Bootstrap complete.
I20240503 13:58:52.389335 4892 ts_tablet_manager.cc:1367] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea: Time spent bootstrapping tablet: real 0.047s user 0.032s sys 0.000s
I20240503 13:58:52.391317 4895 tablet_bootstrap.cc:492] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:58:52.391275 4892 raft_consensus.cc:348] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: true } }
I20240503 13:58:52.391526 4892 raft_consensus.cc:729] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e488cf84f7b74419a218d3512e237fea, State: Initialized, Role: LEARNER
I20240503 13:58:52.391607 4895 tablet_bootstrap.cc:492] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6: Bootstrap complete.
I20240503 13:58:52.391866 4895 ts_tablet_manager.cc:1367] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6: Time spent bootstrapping tablet: real 0.049s user 0.034s sys 0.000s
I20240503 13:58:52.391839 4892 consensus_queue.cc:260] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: true } }
I20240503 13:58:52.392663 4892 ts_tablet_manager.cc:1398] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240503 13:58:52.394086 4895 raft_consensus.cc:348] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: true } }
I20240503 13:58:52.394318 4895 raft_consensus.cc:729] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9b3cf5210e6b4f2188e66a80dca823e6, State: Initialized, Role: LEARNER
I20240503 13:58:52.394485 3084 tablet_copy_service.cc:342] P 694cd43f32a746e7ab0e91db5785ba52: Request end of tablet copy session e488cf84f7b74419a218d3512e237fea-d899ba5a20f0466da4e0e3fc9831e5f5 received from {username='slave'} at 127.0.99.132:49745
I20240503 13:58:52.394560 3084 tablet_copy_service.cc:433] P 694cd43f32a746e7ab0e91db5785ba52: ending tablet copy session e488cf84f7b74419a218d3512e237fea-d899ba5a20f0466da4e0e3fc9831e5f5 on tablet d899ba5a20f0466da4e0e3fc9831e5f5 with peer e488cf84f7b74419a218d3512e237fea
I20240503 13:58:52.394618 4895 consensus_queue.cc:260] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: true } }
I20240503 13:58:52.395514 4895 ts_tablet_manager.cc:1398] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 13:58:52.398811 2952 tablet_copy_service.cc:342] P a9c26918b0724416878324492440c93d: Request end of tablet copy session 9b3cf5210e6b4f2188e66a80dca823e6-ce166608b56540d89bddc4919356ebe7 received from {username='slave'} at 127.0.99.131:38483
I20240503 13:58:52.398900 2952 tablet_copy_service.cc:433] P a9c26918b0724416878324492440c93d: ending tablet copy session 9b3cf5210e6b4f2188e66a80dca823e6-ce166608b56540d89bddc4919356ebe7 on tablet ce166608b56540d89bddc4919356ebe7 with peer 9b3cf5210e6b4f2188e66a80dca823e6
I20240503 13:58:52.402000 4879 tablet_bootstrap.cc:492] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:58:52.402257 4879 tablet_bootstrap.cc:492] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316: Bootstrap complete.
I20240503 13:58:52.402422 4879 ts_tablet_manager.cc:1367] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316: Time spent bootstrapping tablet: real 0.034s user 0.018s sys 0.002s
I20240503 13:58:52.402967 4879 raft_consensus.cc:348] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: true } }
I20240503 13:58:52.403088 4879 raft_consensus.cc:729] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5c33992e399d4e63a2cc07cb4a479316, State: Initialized, Role: LEARNER
I20240503 13:58:52.403185 4879 consensus_queue.cc:260] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: true } }
I20240503 13:58:52.403455 4879 ts_tablet_manager.cc:1398] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s
I20240503 13:58:52.403911 2952 tablet_copy_service.cc:342] P a9c26918b0724416878324492440c93d: Request end of tablet copy session 5c33992e399d4e63a2cc07cb4a479316-1e949fd674614e29a933cdf304d0ee94 received from {username='slave'} at 127.0.99.135:51983
I20240503 13:58:52.403976 2952 tablet_copy_service.cc:433] P a9c26918b0724416878324492440c93d: ending tablet copy session 5c33992e399d4e63a2cc07cb4a479316-1e949fd674614e29a933cdf304d0ee94 on tablet 1e949fd674614e29a933cdf304d0ee94 with peer 5c33992e399d4e63a2cc07cb4a479316
I20240503 13:58:52.407773 4906 tablet_bootstrap.cc:492] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:58:52.408003 4906 tablet_bootstrap.cc:492] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4: Bootstrap complete.
I20240503 13:58:52.408164 4906 ts_tablet_manager.cc:1367] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4: Time spent bootstrapping tablet: real 0.047s user 0.027s sys 0.000s
I20240503 13:58:52.408679 4906 raft_consensus.cc:348] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: true } }
I20240503 13:58:52.408789 4906 raft_consensus.cc:729] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: acc5e87f231a46f5beb89cc24cbe28c4, State: Initialized, Role: LEARNER
I20240503 13:58:52.408888 4906 consensus_queue.cc:260] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: true } }
I20240503 13:58:52.409121 4906 ts_tablet_manager.cc:1398] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:58:52.409660 3084 tablet_copy_service.cc:342] P 694cd43f32a746e7ab0e91db5785ba52: Request end of tablet copy session acc5e87f231a46f5beb89cc24cbe28c4-fa9ffe2aeedf42e89d4e00ac12fb9e92 received from {username='slave'} at 127.0.99.133:54557
I20240503 13:58:52.409752 3084 tablet_copy_service.cc:433] P 694cd43f32a746e7ab0e91db5785ba52: ending tablet copy session acc5e87f231a46f5beb89cc24cbe28c4-fa9ffe2aeedf42e89d4e00ac12fb9e92 on tablet fa9ffe2aeedf42e89d4e00ac12fb9e92 with peer acc5e87f231a46f5beb89cc24cbe28c4
I20240503 13:58:52.423974 4892 ts_tablet_manager.cc:902] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea: Initiating tablet copy from peer 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130:34403)
I20240503 13:58:52.424381 4892 tablet_copy_client.cc:308] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34403
I20240503 13:58:52.424796 3084 tablet_copy_service.cc:140] P 694cd43f32a746e7ab0e91db5785ba52: Received BeginTabletCopySession request for tablet 0f599173c7624329af379768658f6041 from peer e488cf84f7b74419a218d3512e237fea ({username='slave'} at 127.0.99.132:49745)
I20240503 13:58:52.424891 3084 tablet_copy_service.cc:161] P 694cd43f32a746e7ab0e91db5785ba52: Beginning new tablet copy session on tablet 0f599173c7624329af379768658f6041 from peer e488cf84f7b74419a218d3512e237fea at {username='slave'} at 127.0.99.132:49745: session id = e488cf84f7b74419a218d3512e237fea-0f599173c7624329af379768658f6041
I20240503 13:58:52.425858 3084 tablet_copy_source_session.cc:205] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:58:52.426638 4892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f599173c7624329af379768658f6041. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:52.428660 4892 tablet_copy_client.cc:783] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea: tablet copy: Starting download of 0 data blocks...
I20240503 13:58:52.428802 4892 tablet_copy_client.cc:647] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea: tablet copy: Starting download of 1 WAL segments...
I20240503 13:58:52.429828 4892 tablet_copy_client.cc:523] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:58:52.431156 4892 tablet_bootstrap.cc:492] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea: Bootstrap starting.
I20240503 13:58:52.451848 4892 tablet_bootstrap.cc:492] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:58:52.452147 4892 tablet_bootstrap.cc:492] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea: Bootstrap complete.
I20240503 13:58:52.452306 4892 ts_tablet_manager.cc:1367] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea: Time spent bootstrapping tablet: real 0.021s user 0.020s sys 0.000s
I20240503 13:58:52.452868 4892 raft_consensus.cc:348] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: true } }
I20240503 13:58:52.452986 4892 raft_consensus.cc:729] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e488cf84f7b74419a218d3512e237fea, State: Initialized, Role: LEARNER
I20240503 13:58:52.453094 4892 consensus_queue.cc:260] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: true } }
I20240503 13:58:52.453439 4892 ts_tablet_manager.cc:1398] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:58:52.454069 3084 tablet_copy_service.cc:342] P 694cd43f32a746e7ab0e91db5785ba52: Request end of tablet copy session e488cf84f7b74419a218d3512e237fea-0f599173c7624329af379768658f6041 received from {username='slave'} at 127.0.99.132:49745
I20240503 13:58:52.454185 3084 tablet_copy_service.cc:433] P 694cd43f32a746e7ab0e91db5785ba52: ending tablet copy session e488cf84f7b74419a218d3512e237fea-0f599173c7624329af379768658f6041 on tablet 0f599173c7624329af379768658f6041 with peer e488cf84f7b74419a218d3512e237fea
I20240503 13:58:52.709010 4581 raft_consensus.cc:1203] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:58:52.733773 4450 raft_consensus.cc:1203] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:58:52.736085 4319 raft_consensus.cc:1203] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:58:52.762485 4188 raft_consensus.cc:1203] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:58:52.777025 4057 raft_consensus.cc:1203] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:58:52.807775 4450 raft_consensus.cc:1203] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:58:52.817135 4188 raft_consensus.cc:1203] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:58:52.828509 4319 raft_consensus.cc:1203] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:58:52.922529 4581 raft_consensus.cc:1203] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:58:53.145303 5021 raft_consensus.cc:1050] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52: attempting to promote NON_VOTER acc5e87f231a46f5beb89cc24cbe28c4 to VOTER
I20240503 13:58:53.145838 5021 consensus_queue.cc:237] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: false } }
I20240503 13:58:53.147063 4319 raft_consensus.cc:1261] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4 [term 1 LEARNER]: Refusing update from remote peer 694cd43f32a746e7ab0e91db5785ba52: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:58:53.147531 5022 consensus_queue.cc:1035] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:58:53.150110 5021 raft_consensus.cc:2932] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, acc5e87f231a46f5beb89cc24cbe28c4 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: false } } }
I20240503 13:58:53.151695 4319 raft_consensus.cc:2932] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, acc5e87f231a46f5beb89cc24cbe28c4 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: false } } }
I20240503 13:58:53.152451 2782 catalog_manager.cc:5484] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 reported cstate change: config changed from index 4 to 5, acc5e87f231a46f5beb89cc24cbe28c4 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "694cd43f32a746e7ab0e91db5785ba52" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:53.174484 3064 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fa9ffe2aeedf42e89d4e00ac12fb9e92"
dest_uuid: "694cd43f32a746e7ab0e91db5785ba52"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39978
I20240503 13:58:53.174661 3064 raft_consensus.cc:595] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:58:53.201267 5021 raft_consensus.cc:1050] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52: attempting to promote NON_VOTER e488cf84f7b74419a218d3512e237fea to VOTER
I20240503 13:58:53.201645 5021 consensus_queue.cc:237] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: false } }
I20240503 13:58:53.202771 4188 raft_consensus.cc:1261] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea [term 1 LEARNER]: Refusing update from remote peer 694cd43f32a746e7ab0e91db5785ba52: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:58:53.211396 4718 consensus_queue.cc:1035] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:58:53.214461 5022 raft_consensus.cc:2932] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e488cf84f7b74419a218d3512e237fea (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: false } } }
I20240503 13:58:53.215070 4188 raft_consensus.cc:2932] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e488cf84f7b74419a218d3512e237fea (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: false } } }
I20240503 13:58:53.217105 2789 catalog_manager.cc:5484] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 reported cstate change: config changed from index 4 to 5, e488cf84f7b74419a218d3512e237fea (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "694cd43f32a746e7ab0e91db5785ba52" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:53.219362 5020 raft_consensus.cc:1050] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d: attempting to promote NON_VOTER 5c33992e399d4e63a2cc07cb4a479316 to VOTER
I20240503 13:58:53.219920 5020 consensus_queue.cc:237] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: false } }
I20240503 13:58:53.221069 4581 raft_consensus.cc:1261] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316 [term 1 LEARNER]: Refusing update from remote peer a9c26918b0724416878324492440c93d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:58:53.221490 4731 consensus_queue.cc:1035] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:58:53.224287 5020 raft_consensus.cc:2932] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5c33992e399d4e63a2cc07cb4a479316 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: false } } }
I20240503 13:58:53.226680 2782 catalog_manager.cc:5484] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d reported cstate change: config changed from index 4 to 5, 5c33992e399d4e63a2cc07cb4a479316 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a9c26918b0724416878324492440c93d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:53.227914 4581 raft_consensus.cc:2932] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5c33992e399d4e63a2cc07cb4a479316 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: false } } }
I20240503 13:58:53.256090 4731 raft_consensus.cc:1050] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d: attempting to promote NON_VOTER 3b1cfcd5138346b6ac5b8581c536034d to VOTER
I20240503 13:58:53.256417 4731 consensus_queue.cc:237] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: false } }
I20240503 13:58:53.257491 4450 raft_consensus.cc:1261] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d [term 1 LEARNER]: Refusing update from remote peer a9c26918b0724416878324492440c93d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:58:53.257877 4731 consensus_queue.cc:1035] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:58:53.260195 5020 raft_consensus.cc:2932] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3b1cfcd5138346b6ac5b8581c536034d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: false } } }
I20240503 13:58:53.260732 4450 raft_consensus.cc:2932] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3b1cfcd5138346b6ac5b8581c536034d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: false } } }
I20240503 13:58:53.262590 2782 catalog_manager.cc:5484] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d reported cstate change: config changed from index 4 to 5, 3b1cfcd5138346b6ac5b8581c536034d (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a9c26918b0724416878324492440c93d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:53.287767 5021 raft_consensus.cc:1050] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52: attempting to promote NON_VOTER e488cf84f7b74419a218d3512e237fea to VOTER
I20240503 13:58:53.288095 5021 consensus_queue.cc:237] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: false } }
I20240503 13:58:53.289068 4188 raft_consensus.cc:1261] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea [term 1 LEARNER]: Refusing update from remote peer 694cd43f32a746e7ab0e91db5785ba52: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:58:53.289415 5021 consensus_queue.cc:1035] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:58:53.291014 5022 raft_consensus.cc:2932] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e488cf84f7b74419a218d3512e237fea (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: false } } }
I20240503 13:58:53.291404 4188 raft_consensus.cc:2932] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e488cf84f7b74419a218d3512e237fea (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: false } } }
I20240503 13:58:53.293577 2789 catalog_manager.cc:5484] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 reported cstate change: config changed from index 4 to 5, e488cf84f7b74419a218d3512e237fea (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "694cd43f32a746e7ab0e91db5785ba52" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:53.294085 5019 raft_consensus.cc:1050] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d: attempting to promote NON_VOTER 9b3cf5210e6b4f2188e66a80dca823e6 to VOTER
I20240503 13:58:53.294350 5019 consensus_queue.cc:237] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: false } }
I20240503 13:58:53.295509 4057 raft_consensus.cc:1261] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6 [term 1 LEARNER]: Refusing update from remote peer a9c26918b0724416878324492440c93d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:58:53.296012 4731 consensus_queue.cc:1035] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:58:53.298538 5019 raft_consensus.cc:2932] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9b3cf5210e6b4f2188e66a80dca823e6 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: false } } }
I20240503 13:58:53.299017 4057 raft_consensus.cc:2932] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9b3cf5210e6b4f2188e66a80dca823e6 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: false } } }
I20240503 13:58:53.300666 2782 catalog_manager.cc:5484] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d reported cstate change: config changed from index 4 to 5, 9b3cf5210e6b4f2188e66a80dca823e6 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a9c26918b0724416878324492440c93d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:53.350024 5021 raft_consensus.cc:1050] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52: attempting to promote NON_VOTER acc5e87f231a46f5beb89cc24cbe28c4 to VOTER
I20240503 13:58:53.350324 5021 consensus_queue.cc:237] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: false } }
I20240503 13:58:53.351430 4319 raft_consensus.cc:1261] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4 [term 1 LEARNER]: Refusing update from remote peer 694cd43f32a746e7ab0e91db5785ba52: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:58:53.351809 5021 consensus_queue.cc:1035] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:58:53.353466 5022 raft_consensus.cc:2932] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, acc5e87f231a46f5beb89cc24cbe28c4 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: false } } }
I20240503 13:58:53.353899 4319 raft_consensus.cc:2932] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, acc5e87f231a46f5beb89cc24cbe28c4 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: false } } }
I20240503 13:58:53.355733 2789 catalog_manager.cc:5484] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 reported cstate change: config changed from index 4 to 5, acc5e87f231a46f5beb89cc24cbe28c4 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "694cd43f32a746e7ab0e91db5785ba52" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:53.360028 4731 raft_consensus.cc:1050] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d: attempting to promote NON_VOTER 3b1cfcd5138346b6ac5b8581c536034d to VOTER
I20240503 13:58:53.360286 4731 consensus_queue.cc:237] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: false } }
I20240503 13:58:53.361196 4450 raft_consensus.cc:1261] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d [term 1 LEARNER]: Refusing update from remote peer a9c26918b0724416878324492440c93d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:58:53.361598 4731 consensus_queue.cc:1035] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:58:53.363195 5020 raft_consensus.cc:2932] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3b1cfcd5138346b6ac5b8581c536034d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: false } } }
I20240503 13:58:53.363639 4450 raft_consensus.cc:2932] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3b1cfcd5138346b6ac5b8581c536034d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: false } } }
I20240503 13:58:53.365245 2782 catalog_manager.cc:5484] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d reported cstate change: config changed from index 4 to 5, 3b1cfcd5138346b6ac5b8581c536034d (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a9c26918b0724416878324492440c93d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:53.410419 5019 raft_consensus.cc:1050] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d: attempting to promote NON_VOTER 5c33992e399d4e63a2cc07cb4a479316 to VOTER
I20240503 13:58:53.410790 5019 consensus_queue.cc:237] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: false } }
I20240503 13:58:53.411944 4581 raft_consensus.cc:1261] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316 [term 1 LEARNER]: Refusing update from remote peer a9c26918b0724416878324492440c93d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:58:53.412391 5019 consensus_queue.cc:1035] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:58:53.414103 5020 raft_consensus.cc:2932] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5c33992e399d4e63a2cc07cb4a479316 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: false } } }
I20240503 13:58:53.414566 4581 raft_consensus.cc:2932] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5c33992e399d4e63a2cc07cb4a479316 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: false } } }
I20240503 13:58:53.415823 3064 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d899ba5a20f0466da4e0e3fc9831e5f5"
dest_uuid: "694cd43f32a746e7ab0e91db5785ba52"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40024
I20240503 13:58:53.415949 3064 raft_consensus.cc:595] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:58:53.416430 2789 catalog_manager.cc:5484] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d reported cstate change: config changed from index 4 to 5, 5c33992e399d4e63a2cc07cb4a479316 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a9c26918b0724416878324492440c93d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:53.426327 2929 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "25a1559451b8472482ac6e482da41d28"
dest_uuid: "a9c26918b0724416878324492440c93d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42994
I20240503 13:58:53.426473 2929 raft_consensus.cc:595] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Received request to transfer leadership
I20240503 13:58:53.435195 2929 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "03e82ebc742b46ce98fc9ab599b08f46"
dest_uuid: "a9c26918b0724416878324492440c93d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43006
I20240503 13:58:53.435328 2929 raft_consensus.cc:595] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Received request to transfer leadership
I20240503 13:58:53.444393 2929 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "558e844e3d344fcbb0e41cdf132c880c"
dest_uuid: "a9c26918b0724416878324492440c93d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43014
I20240503 13:58:53.444501 2929 raft_consensus.cc:595] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [term 1 LEADER]: Received request to transfer leadership
I20240503 13:58:53.453735 3064 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fa9ffe2aeedf42e89d4e00ac12fb9e92"
dest_uuid: "694cd43f32a746e7ab0e91db5785ba52"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40038
I20240503 13:58:53.453856 3064 raft_consensus.cc:595] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:58:53.463450 2929 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ce166608b56540d89bddc4919356ebe7"
dest_uuid: "a9c26918b0724416878324492440c93d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43024
I20240503 13:58:53.463572 2929 raft_consensus.cc:595] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Received request to transfer leadership
I20240503 13:58:53.473284 3064 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1a88836924b04bfea8524184d061a19b"
dest_uuid: "694cd43f32a746e7ab0e91db5785ba52"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40064
I20240503 13:58:53.473418 3064 raft_consensus.cc:595] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:58:53.482446 2929 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1e949fd674614e29a933cdf304d0ee94"
dest_uuid: "a9c26918b0724416878324492440c93d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43052
I20240503 13:58:53.482564 2929 raft_consensus.cc:595] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Received request to transfer leadership
I20240503 13:58:53.491633 3064 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0f599173c7624329af379768658f6041"
dest_uuid: "694cd43f32a746e7ab0e91db5785ba52"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40088
I20240503 13:58:53.491761 3064 raft_consensus.cc:595] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:58:53.605907 5022 raft_consensus.cc:979] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52: : Instructing follower acc5e87f231a46f5beb89cc24cbe28c4 to start an election
I20240503 13:58:53.606047 5022 raft_consensus.cc:1067] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Signalling peer acc5e87f231a46f5beb89cc24cbe28c4 to start an election
I20240503 13:58:53.606614 4319 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fa9ffe2aeedf42e89d4e00ac12fb9e92"
dest_uuid: "acc5e87f231a46f5beb89cc24cbe28c4"
from {username='slave'} at 127.0.99.130:60381
I20240503 13:58:53.606848 4319 raft_consensus.cc:482] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:58:53.606971 4319 raft_consensus.cc:3037] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:58:53.607875 4319 raft_consensus.cc:504] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: false } }
I20240503 13:58:53.608736 4319 leader_election.cc:288] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4 [CANDIDATE]: Term 2 election: Requested vote from peers 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130:34403)
I20240503 13:58:53.609161 3064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9ffe2aeedf42e89d4e00ac12fb9e92" candidate_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "694cd43f32a746e7ab0e91db5785ba52"
I20240503 13:58:53.609314 3064 raft_consensus.cc:3032] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:58:53.609350 3064 raft_consensus.cc:729] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 694cd43f32a746e7ab0e91db5785ba52, State: Running, Role: LEADER
I20240503 13:58:53.609488 3064 consensus_queue.cc:260] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: false } }
I20240503 13:58:53.609637 3064 raft_consensus.cc:3037] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:58:53.610353 3064 raft_consensus.cc:2454] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc5e87f231a46f5beb89cc24cbe28c4 in term 2.
I20240503 13:58:53.610682 4253 leader_election.cc:302] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 694cd43f32a746e7ab0e91db5785ba52, acc5e87f231a46f5beb89cc24cbe28c4; no voters:
I20240503 13:58:53.610898 4935 raft_consensus.cc:2781] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:58:53.611089 4935 raft_consensus.cc:686] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4 [term 2 LEADER]: Becoming Leader. State: Replica: acc5e87f231a46f5beb89cc24cbe28c4, State: Running, Role: LEADER
I20240503 13:58:53.611397 4935 consensus_queue.cc:237] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: false } }
I20240503 13:58:53.613168 2782 catalog_manager.cc:5484] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4 reported cstate change: term changed from 1 to 2, leader changed from 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130) to acc5e87f231a46f5beb89cc24cbe28c4 (127.0.99.133). New cstate: current_term: 2 leader_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 13:58:53.618834 2775 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet fa9ffe2aeedf42e89d4e00ac12fb9e92 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader acc5e87f231a46f5beb89cc24cbe28c4 (127.0.99.133:45407) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240503 13:58:53.649078 5020 raft_consensus.cc:979] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d: : Instructing follower 5c33992e399d4e63a2cc07cb4a479316 to start an election
I20240503 13:58:53.649197 5020 raft_consensus.cc:1067] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Signalling peer 5c33992e399d4e63a2cc07cb4a479316 to start an election
I20240503 13:58:53.649730 4581 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "25a1559451b8472482ac6e482da41d28"
dest_uuid: "5c33992e399d4e63a2cc07cb4a479316"
from {username='slave'} at 127.0.99.129:42937
I20240503 13:58:53.649906 4581 raft_consensus.cc:482] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:58:53.650008 4581 raft_consensus.cc:3037] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:58:53.650831 4581 raft_consensus.cc:504] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: false } }
I20240503 13:58:53.651729 4581 leader_election.cc:288] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316 [CANDIDATE]: Term 2 election: Requested vote from peers a9c26918b0724416878324492440c93d (127.0.99.129:39461)
I20240503 13:58:53.652153 2929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25a1559451b8472482ac6e482da41d28" candidate_uuid: "5c33992e399d4e63a2cc07cb4a479316" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a9c26918b0724416878324492440c93d"
I20240503 13:58:53.652293 2929 raft_consensus.cc:3032] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:58:53.652328 2929 raft_consensus.cc:729] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a9c26918b0724416878324492440c93d, State: Running, Role: LEADER
I20240503 13:58:53.652460 2929 consensus_queue.cc:260] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: false } }
I20240503 13:58:53.652614 2929 raft_consensus.cc:3037] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:58:53.653350 2929 raft_consensus.cc:2454] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c33992e399d4e63a2cc07cb4a479316 in term 2.
I20240503 13:58:53.653713 4514 leader_election.cc:302] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5c33992e399d4e63a2cc07cb4a479316, a9c26918b0724416878324492440c93d; no voters:
I20240503 13:58:53.653925 4912 raft_consensus.cc:2781] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:58:53.654124 4912 raft_consensus.cc:686] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316 [term 2 LEADER]: Becoming Leader. State: Replica: 5c33992e399d4e63a2cc07cb4a479316, State: Running, Role: LEADER
I20240503 13:58:53.654395 4912 consensus_queue.cc:237] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: false } }
I20240503 13:58:53.656212 2782 catalog_manager.cc:5484] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316 reported cstate change: term changed from 1 to 2, leader changed from a9c26918b0724416878324492440c93d (127.0.99.129) to 5c33992e399d4e63a2cc07cb4a479316 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "5c33992e399d4e63a2cc07cb4a479316" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:53.702378 3064 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d899ba5a20f0466da4e0e3fc9831e5f5"
dest_uuid: "694cd43f32a746e7ab0e91db5785ba52"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40118
I20240503 13:58:53.702530 3064 raft_consensus.cc:595] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:58:53.717806 2929 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "03e82ebc742b46ce98fc9ab599b08f46"
dest_uuid: "a9c26918b0724416878324492440c93d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43076
I20240503 13:58:53.717963 2929 raft_consensus.cc:595] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Received request to transfer leadership
I20240503 13:58:53.727644 2929 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "558e844e3d344fcbb0e41cdf132c880c"
dest_uuid: "a9c26918b0724416878324492440c93d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43094
I20240503 13:58:53.727772 2929 raft_consensus.cc:595] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [term 1 LEADER]: Received request to transfer leadership
I20240503 13:58:53.741601 2929 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ce166608b56540d89bddc4919356ebe7"
dest_uuid: "a9c26918b0724416878324492440c93d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43116
I20240503 13:58:53.741734 2929 raft_consensus.cc:595] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Received request to transfer leadership
I20240503 13:58:53.751029 3064 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1a88836924b04bfea8524184d061a19b"
dest_uuid: "694cd43f32a746e7ab0e91db5785ba52"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40126
I20240503 13:58:53.751156 3064 raft_consensus.cc:595] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:58:53.753866 5022 raft_consensus.cc:979] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52: : Instructing follower e488cf84f7b74419a218d3512e237fea to start an election
I20240503 13:58:53.753959 5022 raft_consensus.cc:1067] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Signalling peer e488cf84f7b74419a218d3512e237fea to start an election
I20240503 13:58:53.754484 4188 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0f599173c7624329af379768658f6041"
dest_uuid: "e488cf84f7b74419a218d3512e237fea"
from {username='slave'} at 127.0.99.130:47129
I20240503 13:58:53.754700 4188 raft_consensus.cc:482] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:58:53.754812 4188 raft_consensus.cc:3037] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:58:53.755848 4188 raft_consensus.cc:504] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: false } }
I20240503 13:58:53.756822 4188 leader_election.cc:288] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea [CANDIDATE]: Term 2 election: Requested vote from peers 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130:34403)
I20240503 13:58:53.757333 3064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f599173c7624329af379768658f6041" candidate_uuid: "e488cf84f7b74419a218d3512e237fea" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "694cd43f32a746e7ab0e91db5785ba52"
I20240503 13:58:53.757452 3064 raft_consensus.cc:3032] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:58:53.757506 3064 raft_consensus.cc:729] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 694cd43f32a746e7ab0e91db5785ba52, State: Running, Role: LEADER
I20240503 13:58:53.757623 5021 raft_consensus.cc:979] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52: : Instructing follower e488cf84f7b74419a218d3512e237fea to start an election
I20240503 13:58:53.757647 3064 consensus_queue.cc:260] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: false } }
I20240503 13:58:53.757753 5021 raft_consensus.cc:1067] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Signalling peer e488cf84f7b74419a218d3512e237fea to start an election
I20240503 13:58:53.757800 3064 raft_consensus.cc:3037] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:58:53.758117 4187 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d899ba5a20f0466da4e0e3fc9831e5f5"
dest_uuid: "e488cf84f7b74419a218d3512e237fea"
from {username='slave'} at 127.0.99.130:47129
I20240503 13:58:53.758219 4187 raft_consensus.cc:482] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:58:53.758287 4187 raft_consensus.cc:3037] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:58:53.758760 3064 raft_consensus.cc:2454] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e488cf84f7b74419a218d3512e237fea in term 2.
I20240503 13:58:53.759124 4187 raft_consensus.cc:504] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: false } }
I20240503 13:58:53.759524 4122 leader_election.cc:302] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 694cd43f32a746e7ab0e91db5785ba52, e488cf84f7b74419a218d3512e237fea; no voters:
I20240503 13:58:53.759524 4187 leader_election.cc:288] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea [CANDIDATE]: Term 2 election: Requested vote from peers 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130:34403)
I20240503 13:58:53.759750 4936 raft_consensus.cc:2781] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:58:53.759922 4936 raft_consensus.cc:686] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea [term 2 LEADER]: Becoming Leader. State: Replica: e488cf84f7b74419a218d3512e237fea, State: Running, Role: LEADER
I20240503 13:58:53.760178 4936 consensus_queue.cc:237] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: false } }
I20240503 13:58:53.761821 3064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d899ba5a20f0466da4e0e3fc9831e5f5" candidate_uuid: "e488cf84f7b74419a218d3512e237fea" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "694cd43f32a746e7ab0e91db5785ba52"
I20240503 13:58:53.761934 3064 raft_consensus.cc:3032] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:58:53.761979 3064 raft_consensus.cc:729] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 694cd43f32a746e7ab0e91db5785ba52, State: Running, Role: LEADER
I20240503 13:58:53.761988 2929 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1e949fd674614e29a933cdf304d0ee94"
dest_uuid: "a9c26918b0724416878324492440c93d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43134
I20240503 13:58:53.762109 2929 raft_consensus.cc:595] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Received request to transfer leadership
I20240503 13:58:53.762118 3064 consensus_queue.cc:260] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: false } }
I20240503 13:58:53.762246 3064 raft_consensus.cc:3037] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:58:53.763118 3064 raft_consensus.cc:2454] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e488cf84f7b74419a218d3512e237fea in term 2.
I20240503 13:58:53.763434 4122 leader_election.cc:302] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 694cd43f32a746e7ab0e91db5785ba52, e488cf84f7b74419a218d3512e237fea; no voters:
I20240503 13:58:53.763811 4936 raft_consensus.cc:2781] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:58:53.763904 4936 raft_consensus.cc:686] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea [term 2 LEADER]: Becoming Leader. State: Replica: e488cf84f7b74419a218d3512e237fea, State: Running, Role: LEADER
I20240503 13:58:53.763880 2782 catalog_manager.cc:5484] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea reported cstate change: term changed from 1 to 2, leader changed from 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130) to e488cf84f7b74419a218d3512e237fea (127.0.99.132). New cstate: current_term: 2 leader_uuid: "e488cf84f7b74419a218d3512e237fea" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:53.764029 4936 consensus_queue.cc:237] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: false } }
I20240503 13:58:53.767570 2782 catalog_manager.cc:5484] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea reported cstate change: term changed from 1 to 2, leader changed from 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130) to e488cf84f7b74419a218d3512e237fea (127.0.99.132). New cstate: current_term: 2 leader_uuid: "e488cf84f7b74419a218d3512e237fea" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:53.812641 5019 raft_consensus.cc:979] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d: : Instructing follower 5c33992e399d4e63a2cc07cb4a479316 to start an election
I20240503 13:58:53.812757 5019 raft_consensus.cc:1067] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Signalling peer 5c33992e399d4e63a2cc07cb4a479316 to start an election
I20240503 13:58:53.813226 4581 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e949fd674614e29a933cdf304d0ee94"
dest_uuid: "5c33992e399d4e63a2cc07cb4a479316"
from {username='slave'} at 127.0.99.129:42937
I20240503 13:58:53.813365 4581 raft_consensus.cc:482] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:58:53.813443 4581 raft_consensus.cc:3037] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:58:53.814286 4581 raft_consensus.cc:504] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: false } }
I20240503 13:58:53.814572 4581 leader_election.cc:288] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316 [CANDIDATE]: Term 2 election: Requested vote from peers a9c26918b0724416878324492440c93d (127.0.99.129:39461)
I20240503 13:58:53.814924 2929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e949fd674614e29a933cdf304d0ee94" candidate_uuid: "5c33992e399d4e63a2cc07cb4a479316" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a9c26918b0724416878324492440c93d"
I20240503 13:58:53.815044 2929 raft_consensus.cc:3032] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:58:53.815093 2929 raft_consensus.cc:729] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a9c26918b0724416878324492440c93d, State: Running, Role: LEADER
I20240503 13:58:53.815228 2929 consensus_queue.cc:260] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: false } }
I20240503 13:58:53.815394 2929 raft_consensus.cc:3037] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:58:53.816221 2929 raft_consensus.cc:2454] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c33992e399d4e63a2cc07cb4a479316 in term 2.
I20240503 13:58:53.816481 4514 leader_election.cc:302] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5c33992e399d4e63a2cc07cb4a479316, a9c26918b0724416878324492440c93d; no voters:
I20240503 13:58:53.816659 4912 raft_consensus.cc:2781] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:58:53.816746 4912 raft_consensus.cc:686] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316 [term 2 LEADER]: Becoming Leader. State: Replica: 5c33992e399d4e63a2cc07cb4a479316, State: Running, Role: LEADER
I20240503 13:58:53.816887 4912 consensus_queue.cc:237] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: false } }
I20240503 13:58:53.818333 2782 catalog_manager.cc:5484] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316 reported cstate change: term changed from 1 to 2, leader changed from a9c26918b0724416878324492440c93d (127.0.99.129) to 5c33992e399d4e63a2cc07cb4a479316 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "5c33992e399d4e63a2cc07cb4a479316" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:53.819818 5020 raft_consensus.cc:979] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d: : Instructing follower 3b1cfcd5138346b6ac5b8581c536034d to start an election
I20240503 13:58:53.819902 5020 raft_consensus.cc:1067] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [term 1 LEADER]: Signalling peer 3b1cfcd5138346b6ac5b8581c536034d to start an election
I20240503 13:58:53.820343 4450 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "558e844e3d344fcbb0e41cdf132c880c"
dest_uuid: "3b1cfcd5138346b6ac5b8581c536034d"
from {username='slave'} at 127.0.99.129:34307
I20240503 13:58:53.820525 4450 raft_consensus.cc:482] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:58:53.820641 4450 raft_consensus.cc:3037] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:58:53.821374 4450 raft_consensus.cc:504] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: false } }
I20240503 13:58:53.822190 4450 leader_election.cc:288] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d [CANDIDATE]: Term 2 election: Requested vote from peers a9c26918b0724416878324492440c93d (127.0.99.129:39461)
I20240503 13:58:53.822477 2929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "558e844e3d344fcbb0e41cdf132c880c" candidate_uuid: "3b1cfcd5138346b6ac5b8581c536034d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a9c26918b0724416878324492440c93d"
I20240503 13:58:53.822578 2929 raft_consensus.cc:3032] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:58:53.822611 2929 raft_consensus.cc:729] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a9c26918b0724416878324492440c93d, State: Running, Role: LEADER
I20240503 13:58:53.822728 2929 consensus_queue.cc:260] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: false } }
I20240503 13:58:53.822831 2929 raft_consensus.cc:3037] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:58:53.823724 2929 raft_consensus.cc:2454] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b1cfcd5138346b6ac5b8581c536034d in term 2.
I20240503 13:58:53.824051 4383 leader_election.cc:302] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3b1cfcd5138346b6ac5b8581c536034d, a9c26918b0724416878324492440c93d; no voters:
I20240503 13:58:53.824265 4894 raft_consensus.cc:2781] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:58:53.824448 4894 raft_consensus.cc:686] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d [term 2 LEADER]: Becoming Leader. State: Replica: 3b1cfcd5138346b6ac5b8581c536034d, State: Running, Role: LEADER
I20240503 13:58:53.824723 4894 consensus_queue.cc:237] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: false } }
I20240503 13:58:53.826786 2782 catalog_manager.cc:5484] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d reported cstate change: term changed from 1 to 2, leader changed from a9c26918b0724416878324492440c93d (127.0.99.129) to 3b1cfcd5138346b6ac5b8581c536034d (127.0.99.134). New cstate: current_term: 2 leader_uuid: "3b1cfcd5138346b6ac5b8581c536034d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:53.832378 5019 raft_consensus.cc:979] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d: : Instructing follower 3b1cfcd5138346b6ac5b8581c536034d to start an election
I20240503 13:58:53.832473 5019 raft_consensus.cc:1067] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Signalling peer 3b1cfcd5138346b6ac5b8581c536034d to start an election
I20240503 13:58:53.832863 4450 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "03e82ebc742b46ce98fc9ab599b08f46"
dest_uuid: "3b1cfcd5138346b6ac5b8581c536034d"
from {username='slave'} at 127.0.99.129:34307
I20240503 13:58:53.832953 4450 raft_consensus.cc:482] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:58:53.833034 4450 raft_consensus.cc:3037] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:58:53.833680 4450 raft_consensus.cc:504] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: false } }
I20240503 13:58:53.833909 4450 leader_election.cc:288] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d [CANDIDATE]: Term 2 election: Requested vote from peers a9c26918b0724416878324492440c93d (127.0.99.129:39461)
I20240503 13:58:53.834193 2929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e82ebc742b46ce98fc9ab599b08f46" candidate_uuid: "3b1cfcd5138346b6ac5b8581c536034d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a9c26918b0724416878324492440c93d"
I20240503 13:58:53.834388 2929 raft_consensus.cc:3032] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:58:53.834455 2929 raft_consensus.cc:729] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a9c26918b0724416878324492440c93d, State: Running, Role: LEADER
I20240503 13:58:53.834563 2929 consensus_queue.cc:260] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: false } }
I20240503 13:58:53.834666 2929 raft_consensus.cc:3037] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:58:53.835340 2929 raft_consensus.cc:2454] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b1cfcd5138346b6ac5b8581c536034d in term 2.
I20240503 13:58:53.835635 4383 leader_election.cc:302] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3b1cfcd5138346b6ac5b8581c536034d, a9c26918b0724416878324492440c93d; no voters:
I20240503 13:58:53.835809 4894 raft_consensus.cc:2781] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:58:53.835901 4894 raft_consensus.cc:686] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d [term 2 LEADER]: Becoming Leader. State: Replica: 3b1cfcd5138346b6ac5b8581c536034d, State: Running, Role: LEADER
I20240503 13:58:53.836035 4894 consensus_queue.cc:237] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: false } }
I20240503 13:58:53.837186 2782 catalog_manager.cc:5484] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d reported cstate change: term changed from 1 to 2, leader changed from a9c26918b0724416878324492440c93d (127.0.99.129) to 3b1cfcd5138346b6ac5b8581c536034d (127.0.99.134). New cstate: current_term: 2 leader_uuid: "3b1cfcd5138346b6ac5b8581c536034d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:53.844746 5019 raft_consensus.cc:979] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d: : Instructing follower 9b3cf5210e6b4f2188e66a80dca823e6 to start an election
I20240503 13:58:53.844830 5019 raft_consensus.cc:1067] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Signalling peer 9b3cf5210e6b4f2188e66a80dca823e6 to start an election
I20240503 13:58:53.845244 4057 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ce166608b56540d89bddc4919356ebe7"
dest_uuid: "9b3cf5210e6b4f2188e66a80dca823e6"
from {username='slave'} at 127.0.99.129:49169
I20240503 13:58:53.845402 4057 raft_consensus.cc:482] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:58:53.845502 4057 raft_consensus.cc:3037] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:58:53.846195 4057 raft_consensus.cc:504] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: false } }
I20240503 13:58:53.846973 4057 leader_election.cc:288] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6 [CANDIDATE]: Term 2 election: Requested vote from peers a9c26918b0724416878324492440c93d (127.0.99.129:39461)
I20240503 13:58:53.847276 2929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce166608b56540d89bddc4919356ebe7" candidate_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a9c26918b0724416878324492440c93d"
I20240503 13:58:53.847404 2929 raft_consensus.cc:3032] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:58:53.847436 2929 raft_consensus.cc:729] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a9c26918b0724416878324492440c93d, State: Running, Role: LEADER
I20240503 13:58:53.847576 2929 consensus_queue.cc:260] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: false } }
I20240503 13:58:53.847687 2929 raft_consensus.cc:3037] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:58:53.848278 2929 raft_consensus.cc:2454] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b3cf5210e6b4f2188e66a80dca823e6 in term 2.
I20240503 13:58:53.848559 3990 leader_election.cc:302] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 9b3cf5210e6b4f2188e66a80dca823e6, a9c26918b0724416878324492440c93d; no voters:
I20240503 13:58:53.848876 5315 raft_consensus.cc:2781] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:58:53.849061 5315 raft_consensus.cc:686] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6 [term 2 LEADER]: Becoming Leader. State: Replica: 9b3cf5210e6b4f2188e66a80dca823e6, State: Running, Role: LEADER
I20240503 13:58:53.849318 5315 consensus_queue.cc:237] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: false } }
I20240503 13:58:53.850919 2782 catalog_manager.cc:5484] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6 reported cstate change: term changed from 1 to 2, leader changed from a9c26918b0724416878324492440c93d (127.0.99.129) to 9b3cf5210e6b4f2188e66a80dca823e6 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:53.890061 4718 raft_consensus.cc:979] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52: : Instructing follower acc5e87f231a46f5beb89cc24cbe28c4 to start an election
I20240503 13:58:53.890167 4718 raft_consensus.cc:1067] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Signalling peer acc5e87f231a46f5beb89cc24cbe28c4 to start an election
I20240503 13:58:53.890561 4319 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1a88836924b04bfea8524184d061a19b"
dest_uuid: "acc5e87f231a46f5beb89cc24cbe28c4"
from {username='slave'} at 127.0.99.130:60381
I20240503 13:58:53.890686 4319 raft_consensus.cc:482] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:58:53.890750 4319 raft_consensus.cc:3037] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:58:53.891559 4319 raft_consensus.cc:504] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: false } }
I20240503 13:58:53.891855 4319 leader_election.cc:288] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4 [CANDIDATE]: Term 2 election: Requested vote from peers 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130:34403)
I20240503 13:58:53.892202 3064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88836924b04bfea8524184d061a19b" candidate_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "694cd43f32a746e7ab0e91db5785ba52"
I20240503 13:58:53.892344 3064 raft_consensus.cc:3032] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:58:53.892393 3064 raft_consensus.cc:729] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 694cd43f32a746e7ab0e91db5785ba52, State: Running, Role: LEADER
I20240503 13:58:53.892539 3064 consensus_queue.cc:260] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: false } }
I20240503 13:58:53.892665 3064 raft_consensus.cc:3037] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:58:53.893374 3064 raft_consensus.cc:2454] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc5e87f231a46f5beb89cc24cbe28c4 in term 2.
I20240503 13:58:53.893651 4253 leader_election.cc:302] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 694cd43f32a746e7ab0e91db5785ba52, acc5e87f231a46f5beb89cc24cbe28c4; no voters:
I20240503 13:58:53.893827 4935 raft_consensus.cc:2781] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:58:53.893936 4935 raft_consensus.cc:686] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4 [term 2 LEADER]: Becoming Leader. State: Replica: acc5e87f231a46f5beb89cc24cbe28c4, State: Running, Role: LEADER
I20240503 13:58:53.894094 4935 consensus_queue.cc:237] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: false } }
I20240503 13:58:53.895609 2782 catalog_manager.cc:5484] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4 reported cstate change: term changed from 1 to 2, leader changed from 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130) to acc5e87f231a46f5beb89cc24cbe28c4 (127.0.99.133). New cstate: current_term: 2 leader_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:54.170523 2929 raft_consensus.cc:1261] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d [term 2 FOLLOWER]: Refusing update from remote peer 5c33992e399d4e63a2cc07cb4a479316: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 13:58:54.171066 4912 consensus_queue.cc:1035] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:58:54.205617 3064 raft_consensus.cc:1261] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [term 2 FOLLOWER]: Refusing update from remote peer acc5e87f231a46f5beb89cc24cbe28c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 13:58:54.206249 4935 consensus_queue.cc:1035] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:58:54.218410 2929 raft_consensus.cc:1261] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [term 2 FOLLOWER]: Refusing update from remote peer 3b1cfcd5138346b6ac5b8581c536034d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 13:58:54.219214 4894 consensus_queue.cc:1035] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:58:54.226540 2929 raft_consensus.cc:1261] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [term 2 FOLLOWER]: Refusing update from remote peer 5c33992e399d4e63a2cc07cb4a479316: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 13:58:54.226939 4912 consensus_queue.cc:1035] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:58:54.246167 4319 consensus_queue.cc:237] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: false } }
I20240503 13:58:54.246754 5231 raft_consensus.cc:2932] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: false } } }
I20240503 13:58:54.248245 2775 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fa9ffe2aeedf42e89d4e00ac12fb9e92 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 13:58:54.249135 2789 catalog_manager.cc:5484] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P acc5e87f231a46f5beb89cc24cbe28c4 reported cstate change: config changed from index 5 to 7, VOTER 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:54.251598 3042 tablet_service.cc:1514] Processing DeleteTablet for tablet fa9ffe2aeedf42e89d4e00ac12fb9e92 with delete_type TABLET_DATA_TOMBSTONED (TS 694cd43f32a746e7ab0e91db5785ba52 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49062
I20240503 13:58:54.252094 5399 tablet_replica.cc:330] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52: stopping tablet replica
I20240503 13:58:54.252437 5399 raft_consensus.cc:2229] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:58:54.252620 5399 raft_consensus.cc:2258] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:58:54.253716 5399 ts_tablet_manager.cc:1875] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:58:54.256009 5399 ts_tablet_manager.cc:1888] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 13:58:54.256103 5399 log.cc:1196] T fa9ffe2aeedf42e89d4e00ac12fb9e92 P 694cd43f32a746e7ab0e91db5785ba52: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/fa9ffe2aeedf42e89d4e00ac12fb9e92
I20240503 13:58:54.256768 2776 catalog_manager.cc:4830] TS 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130:34403): tablet fa9ffe2aeedf42e89d4e00ac12fb9e92 (table rebalance_test_table_3 [id=785472d7e43343c0a92a6afb07b14c83]) successfully deleted
I20240503 13:58:54.268904 3064 raft_consensus.cc:1261] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 [term 2 FOLLOWER]: Refusing update from remote peer e488cf84f7b74419a218d3512e237fea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 13:58:54.269531 5415 consensus_queue.cc:1035] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea [LEADER]: Connected to new peer: Peer: permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:58:54.286456 3064 raft_consensus.cc:1261] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [term 2 FOLLOWER]: Refusing update from remote peer acc5e87f231a46f5beb89cc24cbe28c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 13:58:54.286830 5231 consensus_queue.cc:1035] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:58:54.303948 2929 raft_consensus.cc:1261] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [term 2 FOLLOWER]: Refusing update from remote peer 9b3cf5210e6b4f2188e66a80dca823e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 13:58:54.304423 5315 consensus_queue.cc:1035] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:58:54.337437 4581 consensus_queue.cc:237] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: false } }
I20240503 13:58:54.338106 4912 raft_consensus.cc:2932] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER a9c26918b0724416878324492440c93d (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: false } } }
I20240503 13:58:54.338533 2929 raft_consensus.cc:1261] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [term 2 FOLLOWER]: Refusing update from remote peer 3b1cfcd5138346b6ac5b8581c536034d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 13:58:54.338912 4894 consensus_queue.cc:1035] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:58:54.339535 2777 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 25a1559451b8472482ac6e482da41d28 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 13:58:54.340529 2789 catalog_manager.cc:5484] T 25a1559451b8472482ac6e482da41d28 P 5c33992e399d4e63a2cc07cb4a479316 reported cstate change: config changed from index 5 to 7, VOTER a9c26918b0724416878324492440c93d (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "5c33992e399d4e63a2cc07cb4a479316" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:54.342967 2912 tablet_service.cc:1514] Processing DeleteTablet for tablet 25a1559451b8472482ac6e482da41d28 with delete_type TABLET_DATA_TOMBSTONED (TS a9c26918b0724416878324492440c93d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46422
I20240503 13:58:54.343444 5462 tablet_replica.cc:330] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d: stopping tablet replica
I20240503 13:58:54.343762 5462 raft_consensus.cc:2229] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:58:54.343924 5462 raft_consensus.cc:2258] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:58:54.344820 5462 ts_tablet_manager.cc:1875] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:58:54.345779 3064 raft_consensus.cc:1261] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [term 2 FOLLOWER]: Refusing update from remote peer e488cf84f7b74419a218d3512e237fea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 13:58:54.346195 5415 consensus_queue.cc:1035] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea [LEADER]: Connected to new peer: Peer: permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:58:54.346828 5462 ts_tablet_manager.cc:1888] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 13:58:54.346917 5462 log.cc:1196] T 25a1559451b8472482ac6e482da41d28 P a9c26918b0724416878324492440c93d: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/25a1559451b8472482ac6e482da41d28
I20240503 13:58:54.347707 2775 catalog_manager.cc:4830] TS a9c26918b0724416878324492440c93d (127.0.99.129:39461): tablet 25a1559451b8472482ac6e482da41d28 (table rebalance_test_table_2 [id=0de733125f7b4a63a5162b2d584f7379]) successfully deleted
I20240503 13:58:54.416471 4187 consensus_queue.cc:237] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: false } }
I20240503 13:58:54.417121 5464 raft_consensus.cc:2932] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: false } } }
I20240503 13:58:54.418383 2778 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d899ba5a20f0466da4e0e3fc9831e5f5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 13:58:54.419157 2782 catalog_manager.cc:5484] T d899ba5a20f0466da4e0e3fc9831e5f5 P e488cf84f7b74419a218d3512e237fea reported cstate change: config changed from index 5 to 7, VOTER 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "e488cf84f7b74419a218d3512e237fea" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:54.421481 3042 tablet_service.cc:1514] Processing DeleteTablet for tablet d899ba5a20f0466da4e0e3fc9831e5f5 with delete_type TABLET_DATA_TOMBSTONED (TS 694cd43f32a746e7ab0e91db5785ba52 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49062
I20240503 13:58:54.421744 5399 tablet_replica.cc:330] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52: stopping tablet replica
I20240503 13:58:54.421880 5399 raft_consensus.cc:2229] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:58:54.421988 5399 raft_consensus.cc:2258] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:58:54.422575 5399 ts_tablet_manager.cc:1875] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:58:54.424207 4187 consensus_queue.cc:237] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: false } }
I20240503 13:58:54.424572 5399 ts_tablet_manager.cc:1888] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 13:58:54.424643 5399 log.cc:1196] T d899ba5a20f0466da4e0e3fc9831e5f5 P 694cd43f32a746e7ab0e91db5785ba52: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/d899ba5a20f0466da4e0e3fc9831e5f5
I20240503 13:58:54.424800 5464 raft_consensus.cc:2932] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: false } } }
I20240503 13:58:54.425376 2776 catalog_manager.cc:4830] TS 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130:34403): tablet d899ba5a20f0466da4e0e3fc9831e5f5 (table rebalance_test_table_3 [id=785472d7e43343c0a92a6afb07b14c83]) successfully deleted
I20240503 13:58:54.426143 4450 consensus_queue.cc:237] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: false } }
I20240503 13:58:54.426680 2778 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0f599173c7624329af379768658f6041 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 13:58:54.426700 4894 raft_consensus.cc:2932] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER a9c26918b0724416878324492440c93d (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: false } } }
I20240503 13:58:54.426885 2782 catalog_manager.cc:5484] T 0f599173c7624329af379768658f6041 P e488cf84f7b74419a218d3512e237fea reported cstate change: config changed from index 5 to 7, VOTER 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "e488cf84f7b74419a218d3512e237fea" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e488cf84f7b74419a218d3512e237fea" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:54.428004 2777 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 03e82ebc742b46ce98fc9ab599b08f46 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 13:58:54.428762 2789 catalog_manager.cc:5484] T 03e82ebc742b46ce98fc9ab599b08f46 P 3b1cfcd5138346b6ac5b8581c536034d reported cstate change: config changed from index 5 to 7, VOTER a9c26918b0724416878324492440c93d (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "3b1cfcd5138346b6ac5b8581c536034d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:54.429078 3042 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f599173c7624329af379768658f6041 with delete_type TABLET_DATA_TOMBSTONED (TS 694cd43f32a746e7ab0e91db5785ba52 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49062
I20240503 13:58:54.429509 5399 tablet_replica.cc:330] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52: stopping tablet replica
I20240503 13:58:54.429646 5399 raft_consensus.cc:2229] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:58:54.429744 5399 raft_consensus.cc:2258] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:58:54.429937 4581 consensus_queue.cc:237] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: false } }
I20240503 13:58:54.430254 5399 ts_tablet_manager.cc:1875] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:58:54.430450 4912 raft_consensus.cc:2932] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER a9c26918b0724416878324492440c93d (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: false } } }
I20240503 13:58:54.431676 2912 tablet_service.cc:1514] Processing DeleteTablet for tablet 03e82ebc742b46ce98fc9ab599b08f46 with delete_type TABLET_DATA_TOMBSTONED (TS a9c26918b0724416878324492440c93d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46422
I20240503 13:58:54.431890 5462 tablet_replica.cc:330] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d: stopping tablet replica
I20240503 13:58:54.432037 5462 raft_consensus.cc:2229] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:58:54.432098 2777 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1e949fd674614e29a933cdf304d0ee94 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 13:58:54.432163 5462 raft_consensus.cc:2258] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:58:54.432250 5399 ts_tablet_manager.cc:1888] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 13:58:54.432380 5399 log.cc:1196] T 0f599173c7624329af379768658f6041 P 694cd43f32a746e7ab0e91db5785ba52: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/0f599173c7624329af379768658f6041
I20240503 13:58:54.432577 2789 catalog_manager.cc:5484] T 1e949fd674614e29a933cdf304d0ee94 P 5c33992e399d4e63a2cc07cb4a479316 reported cstate change: config changed from index 5 to 7, VOTER a9c26918b0724416878324492440c93d (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "5c33992e399d4e63a2cc07cb4a479316" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5c33992e399d4e63a2cc07cb4a479316" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:54.432757 5462 ts_tablet_manager.cc:1875] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:58:54.432940 2776 catalog_manager.cc:4830] TS 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130:34403): tablet 0f599173c7624329af379768658f6041 (table rebalance_test_table_1 [id=a1c069f80f8f412494da8d1b96fa3129]) successfully deleted
I20240503 13:58:54.434330 5462 ts_tablet_manager.cc:1888] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 13:58:54.434396 5462 log.cc:1196] T 03e82ebc742b46ce98fc9ab599b08f46 P a9c26918b0724416878324492440c93d: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/03e82ebc742b46ce98fc9ab599b08f46
I20240503 13:58:54.434522 2912 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e949fd674614e29a933cdf304d0ee94 with delete_type TABLET_DATA_TOMBSTONED (TS a9c26918b0724416878324492440c93d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46422
I20240503 13:58:54.434816 5462 tablet_replica.cc:330] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d: stopping tablet replica
I20240503 13:58:54.434880 2775 catalog_manager.cc:4830] TS a9c26918b0724416878324492440c93d (127.0.99.129:39461): tablet 03e82ebc742b46ce98fc9ab599b08f46 (table rebalance_test_table_0 [id=7602235178ee496990dab5c7d3ce3a4a]) successfully deleted
I20240503 13:58:54.434958 5462 raft_consensus.cc:2229] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:58:54.435060 5462 raft_consensus.cc:2258] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:58:54.435263 4450 consensus_queue.cc:237] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: false } }
I20240503 13:58:54.435667 5462 ts_tablet_manager.cc:1875] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:58:54.435827 4894 raft_consensus.cc:2932] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER a9c26918b0724416878324492440c93d (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: false } } }
I20240503 13:58:54.436916 2777 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 558e844e3d344fcbb0e41cdf132c880c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 13:58:54.437201 5462 ts_tablet_manager.cc:1888] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 13:58:54.437276 5462 log.cc:1196] T 1e949fd674614e29a933cdf304d0ee94 P a9c26918b0724416878324492440c93d: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/1e949fd674614e29a933cdf304d0ee94
I20240503 13:58:54.437651 2789 catalog_manager.cc:5484] T 558e844e3d344fcbb0e41cdf132c880c P 3b1cfcd5138346b6ac5b8581c536034d reported cstate change: config changed from index 5 to 7, VOTER a9c26918b0724416878324492440c93d (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "3b1cfcd5138346b6ac5b8581c536034d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3b1cfcd5138346b6ac5b8581c536034d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38729 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:54.437739 2775 catalog_manager.cc:4830] TS a9c26918b0724416878324492440c93d (127.0.99.129:39461): tablet 1e949fd674614e29a933cdf304d0ee94 (table rebalance_test_table_4 [id=3860ccce8f8f4a17b1dc250f91397df8]) successfully deleted
I20240503 13:58:54.439379 2912 tablet_service.cc:1514] Processing DeleteTablet for tablet 558e844e3d344fcbb0e41cdf132c880c with delete_type TABLET_DATA_TOMBSTONED (TS a9c26918b0724416878324492440c93d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46422
I20240503 13:58:54.439560 5462 tablet_replica.cc:330] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d: stopping tablet replica
I20240503 13:58:54.439668 5462 raft_consensus.cc:2229] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:58:54.439769 5462 raft_consensus.cc:2258] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:58:54.440228 5462 ts_tablet_manager.cc:1875] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:58:54.441591 5462 ts_tablet_manager.cc:1888] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 13:58:54.441658 5462 log.cc:1196] T 558e844e3d344fcbb0e41cdf132c880c P a9c26918b0724416878324492440c93d: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/558e844e3d344fcbb0e41cdf132c880c
I20240503 13:58:54.442107 2775 catalog_manager.cc:4830] TS a9c26918b0724416878324492440c93d (127.0.99.129:39461): tablet 558e844e3d344fcbb0e41cdf132c880c (table rebalance_test_table_2 [id=0de733125f7b4a63a5162b2d584f7379]) successfully deleted
I20240503 13:58:54.504547 4319 consensus_queue.cc:237] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: false } }
I20240503 13:58:54.505281 5231 raft_consensus.cc:2932] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: false } } }
I20240503 13:58:54.506567 2775 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1a88836924b04bfea8524184d061a19b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 13:58:54.507494 2789 catalog_manager.cc:5484] T 1a88836924b04bfea8524184d061a19b P acc5e87f231a46f5beb89cc24cbe28c4 reported cstate change: config changed from index 5 to 7, VOTER 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "acc5e87f231a46f5beb89cc24cbe28c4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:54.509739 3042 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a88836924b04bfea8524184d061a19b with delete_type TABLET_DATA_TOMBSTONED (TS 694cd43f32a746e7ab0e91db5785ba52 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49062
I20240503 13:58:54.509963 5399 tablet_replica.cc:330] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52: stopping tablet replica
I20240503 13:58:54.510097 5399 raft_consensus.cc:2229] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:58:54.510208 5399 raft_consensus.cc:2258] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:58:54.510725 5399 ts_tablet_manager.cc:1875] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:58:54.512465 5399 ts_tablet_manager.cc:1888] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 13:58:54.512533 5399 log.cc:1196] T 1a88836924b04bfea8524184d061a19b P 694cd43f32a746e7ab0e91db5785ba52: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/1a88836924b04bfea8524184d061a19b
I20240503 13:58:54.512995 2776 catalog_manager.cc:4830] TS 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130:34403): tablet 1a88836924b04bfea8524184d061a19b (table rebalance_test_table_1 [id=a1c069f80f8f412494da8d1b96fa3129]) successfully deleted
I20240503 13:58:54.513918 4057 consensus_queue.cc:237] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: false } }
I20240503 13:58:54.514518 5315 raft_consensus.cc:2932] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER a9c26918b0724416878324492440c93d (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: false } } }
I20240503 13:58:54.516456 2775 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ce166608b56540d89bddc4919356ebe7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 13:58:54.516654 2789 catalog_manager.cc:5484] T ce166608b56540d89bddc4919356ebe7 P 9b3cf5210e6b4f2188e66a80dca823e6 reported cstate change: config changed from index 5 to 7, VOTER a9c26918b0724416878324492440c93d (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:54.519227 2912 tablet_service.cc:1514] Processing DeleteTablet for tablet ce166608b56540d89bddc4919356ebe7 with delete_type TABLET_DATA_TOMBSTONED (TS a9c26918b0724416878324492440c93d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46422
I20240503 13:58:54.519479 5462 tablet_replica.cc:330] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d: stopping tablet replica
I20240503 13:58:54.519623 5462 raft_consensus.cc:2229] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:58:54.519727 5462 raft_consensus.cc:2258] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:58:54.520367 5462 ts_tablet_manager.cc:1875] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:58:54.522496 5462 ts_tablet_manager.cc:1888] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 13:58:54.522570 5462 log.cc:1196] T ce166608b56540d89bddc4919356ebe7 P a9c26918b0724416878324492440c93d: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ce166608b56540d89bddc4919356ebe7
I20240503 13:58:54.523126 2775 catalog_manager.cc:4830] TS a9c26918b0724416878324492440c93d (127.0.99.129:39461): tablet ce166608b56540d89bddc4919356ebe7 (table rebalance_test_table_0 [id=7602235178ee496990dab5c7d3ce3a4a]) successfully deleted
I20240503 13:58:54.566138 2929 consensus_queue.cc:237] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: true } }
I20240503 13:58:54.567274 5512 raft_consensus.cc:2932] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9b3cf5210e6b4f2188e66a80dca823e6 (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: true } } }
W20240503 13:58:54.567893 2866 consensus_peers.cc:485] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d -> Peer 9b3cf5210e6b4f2188e66a80dca823e6 (127.0.99.131:44145): Couldn't send request to peer 9b3cf5210e6b4f2188e66a80dca823e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5e343562d6043b79af234ed9cdd69d4. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:54.569764 2789 catalog_manager.cc:5484] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d reported cstate change: config changed from index -1 to 4, NON_VOTER 9b3cf5210e6b4f2188e66a80dca823e6 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "a9c26918b0724416878324492440c93d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:55.108116 5530 ts_tablet_manager.cc:902] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6: Initiating tablet copy from peer a9c26918b0724416878324492440c93d (127.0.99.129:39461)
I20240503 13:58:55.108639 5530 tablet_copy_client.cc:308] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:39461
I20240503 13:58:55.109148 2952 tablet_copy_service.cc:140] P a9c26918b0724416878324492440c93d: Received BeginTabletCopySession request for tablet e5e343562d6043b79af234ed9cdd69d4 from peer 9b3cf5210e6b4f2188e66a80dca823e6 ({username='slave'} at 127.0.99.131:38483)
I20240503 13:58:55.109256 2952 tablet_copy_service.cc:161] P a9c26918b0724416878324492440c93d: Beginning new tablet copy session on tablet e5e343562d6043b79af234ed9cdd69d4 from peer 9b3cf5210e6b4f2188e66a80dca823e6 at {username='slave'} at 127.0.99.131:38483: session id = 9b3cf5210e6b4f2188e66a80dca823e6-e5e343562d6043b79af234ed9cdd69d4
I20240503 13:58:55.109956 2952 tablet_copy_source_session.cc:205] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:58:55.110759 5530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5e343562d6043b79af234ed9cdd69d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:55.112428 5530 tablet_copy_client.cc:783] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6: tablet copy: Starting download of 0 data blocks...
I20240503 13:58:55.112587 5530 tablet_copy_client.cc:647] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6: tablet copy: Starting download of 1 WAL segments...
I20240503 13:58:55.113658 5530 tablet_copy_client.cc:523] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:58:55.114588 5530 tablet_bootstrap.cc:492] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6: Bootstrap starting.
I20240503 13:58:55.131747 5530 tablet_bootstrap.cc:492] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:58:55.132027 5530 tablet_bootstrap.cc:492] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6: Bootstrap complete.
I20240503 13:58:55.132189 5530 ts_tablet_manager.cc:1367] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6: Time spent bootstrapping tablet: real 0.018s user 0.017s sys 0.000s
I20240503 13:58:55.132683 5530 raft_consensus.cc:348] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: true } }
I20240503 13:58:55.132789 5530 raft_consensus.cc:729] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9b3cf5210e6b4f2188e66a80dca823e6, State: Initialized, Role: LEARNER
I20240503 13:58:55.132890 5530 consensus_queue.cc:260] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: true } }
I20240503 13:58:55.133219 5530 ts_tablet_manager.cc:1398] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:58:55.133836 2952 tablet_copy_service.cc:342] P a9c26918b0724416878324492440c93d: Request end of tablet copy session 9b3cf5210e6b4f2188e66a80dca823e6-e5e343562d6043b79af234ed9cdd69d4 received from {username='slave'} at 127.0.99.131:38483
I20240503 13:58:55.133934 2952 tablet_copy_service.cc:433] P a9c26918b0724416878324492440c93d: ending tablet copy session 9b3cf5210e6b4f2188e66a80dca823e6-e5e343562d6043b79af234ed9cdd69d4 on tablet e5e343562d6043b79af234ed9cdd69d4 with peer 9b3cf5210e6b4f2188e66a80dca823e6
I20240503 13:58:55.692873 4057 raft_consensus.cc:1203] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:58:56.090970 5553 raft_consensus.cc:1050] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d: attempting to promote NON_VOTER 9b3cf5210e6b4f2188e66a80dca823e6 to VOTER
I20240503 13:58:56.091295 5553 consensus_queue.cc:237] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: false } }
I20240503 13:58:56.092247 4057 raft_consensus.cc:1261] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6 [term 1 LEARNER]: Refusing update from remote peer a9c26918b0724416878324492440c93d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:58:56.092581 5553 consensus_queue.cc:1035] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:58:56.094449 5554 raft_consensus.cc:2932] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9b3cf5210e6b4f2188e66a80dca823e6 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: false } } }
I20240503 13:58:56.094866 4057 raft_consensus.cc:2932] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9b3cf5210e6b4f2188e66a80dca823e6 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: false } } }
I20240503 13:58:56.096859 2782 catalog_manager.cc:5484] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d reported cstate change: config changed from index 4 to 5, 9b3cf5210e6b4f2188e66a80dca823e6 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a9c26918b0724416878324492440c93d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:56.229418 2929 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e5e343562d6043b79af234ed9cdd69d4"
dest_uuid: "a9c26918b0724416878324492440c93d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43216
I20240503 13:58:56.229543 2929 raft_consensus.cc:595] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Received request to transfer leadership
I20240503 13:58:56.440181 2929 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e5e343562d6043b79af234ed9cdd69d4"
dest_uuid: "a9c26918b0724416878324492440c93d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43232
I20240503 13:58:56.440315 2929 raft_consensus.cc:595] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Received request to transfer leadership
I20240503 13:58:56.572021 5553 raft_consensus.cc:979] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d: : Instructing follower 9b3cf5210e6b4f2188e66a80dca823e6 to start an election
I20240503 13:58:56.572151 5553 raft_consensus.cc:1067] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Signalling peer 9b3cf5210e6b4f2188e66a80dca823e6 to start an election
I20240503 13:58:56.572716 4057 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e5e343562d6043b79af234ed9cdd69d4"
dest_uuid: "9b3cf5210e6b4f2188e66a80dca823e6"
from {username='slave'} at 127.0.99.129:49169
I20240503 13:58:56.572876 4057 raft_consensus.cc:482] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:58:56.572938 4057 raft_consensus.cc:3037] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:58:56.574047 4057 raft_consensus.cc:504] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: false } }
I20240503 13:58:56.574374 4057 leader_election.cc:288] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6 [CANDIDATE]: Term 2 election: Requested vote from peers a9c26918b0724416878324492440c93d (127.0.99.129:39461)
I20240503 13:58:56.574903 2929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5e343562d6043b79af234ed9cdd69d4" candidate_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a9c26918b0724416878324492440c93d"
I20240503 13:58:56.575073 2929 raft_consensus.cc:3032] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:58:56.575134 2929 raft_consensus.cc:729] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a9c26918b0724416878324492440c93d, State: Running, Role: LEADER
I20240503 13:58:56.575253 2929 consensus_queue.cc:260] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: false } }
I20240503 13:58:56.575404 2929 raft_consensus.cc:3037] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:58:56.576117 2929 raft_consensus.cc:2454] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b3cf5210e6b4f2188e66a80dca823e6 in term 2.
I20240503 13:58:56.576428 3990 leader_election.cc:302] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 9b3cf5210e6b4f2188e66a80dca823e6, a9c26918b0724416878324492440c93d; no voters:
I20240503 13:58:56.576608 5555 raft_consensus.cc:2781] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:58:56.576764 5555 raft_consensus.cc:686] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6 [term 2 LEADER]: Becoming Leader. State: Replica: 9b3cf5210e6b4f2188e66a80dca823e6, State: Running, Role: LEADER
I20240503 13:58:56.576889 5555 consensus_queue.cc:237] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: false } }
I20240503 13:58:56.578205 2782 catalog_manager.cc:5484] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6 reported cstate change: term changed from 1 to 2, leader changed from a9c26918b0724416878324492440c93d (127.0.99.129) to 9b3cf5210e6b4f2188e66a80dca823e6 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 13:58:56.580747 2775 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet e5e343562d6043b79af234ed9cdd69d4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 9b3cf5210e6b4f2188e66a80dca823e6 (127.0.99.131:44145) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240503 13:58:56.959136 2929 raft_consensus.cc:1261] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [term 2 FOLLOWER]: Refusing update from remote peer 9b3cf5210e6b4f2188e66a80dca823e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 13:58:56.959620 5555 consensus_queue.cc:1035] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:58:57.204334 4057 consensus_queue.cc:237] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: false } }
I20240503 13:58:57.205088 5592 raft_consensus.cc:2932] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER a9c26918b0724416878324492440c93d (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: false } } }
I20240503 13:58:57.206534 2775 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e5e343562d6043b79af234ed9cdd69d4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 13:58:57.207566 2789 catalog_manager.cc:5484] T e5e343562d6043b79af234ed9cdd69d4 P 9b3cf5210e6b4f2188e66a80dca823e6 reported cstate change: config changed from index 5 to 7, VOTER a9c26918b0724416878324492440c93d (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9b3cf5210e6b4f2188e66a80dca823e6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44145 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:57.210080 2912 tablet_service.cc:1514] Processing DeleteTablet for tablet e5e343562d6043b79af234ed9cdd69d4 with delete_type TABLET_DATA_TOMBSTONED (TS a9c26918b0724416878324492440c93d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46422
I20240503 13:58:57.210422 5614 tablet_replica.cc:330] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d: stopping tablet replica
I20240503 13:58:57.210549 5614 raft_consensus.cc:2229] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:58:57.210654 5614 raft_consensus.cc:2258] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:58:57.211247 5614 ts_tablet_manager.cc:1875] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:58:57.213083 5614 ts_tablet_manager.cc:1888] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 13:58:57.213152 5614 log.cc:1196] T e5e343562d6043b79af234ed9cdd69d4 P a9c26918b0724416878324492440c93d: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/e5e343562d6043b79af234ed9cdd69d4
I20240503 13:58:57.213642 2775 catalog_manager.cc:4830] TS a9c26918b0724416878324492440c93d (127.0.99.129:39461): tablet e5e343562d6043b79af234ed9cdd69d4 (table rebalance_test_table_4 [id=3860ccce8f8f4a17b1dc250f91397df8]) successfully deleted
I20240503 13:58:57.319226 4430 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.320848 3042 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:57.322098 4299 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.323390 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.328562 4561 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.336896 2912 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:57.339661 4168 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.422570 3042 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:57.428045 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.429119 4299 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.435032 4430 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.436084 4168 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.441627 2912 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:57.442030 4561 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.664386 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.665884 4430 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.667445 3042 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:57.670647 4168 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.672822 4561 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.685477 4299 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.686224 2912 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:57.794716 3042 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:57.795840 4561 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.797230 4168 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.797932 4430 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.798717 4299 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.803128 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.807550 2912 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:57.912014 4430 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.912415 4561 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.913239 3042 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:57.913553 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.918226 4168 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.924080 2912 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:57.940348 4299 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b0ba17e1ed7e481fa58f7ce6b035ccaa | 127.0.99.190:37103 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:45197 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
3b1cfcd5138346b6ac5b8581c536034d | 127.0.99.134:38729 | HEALTHY | <none> | 2 | 0
5c33992e399d4e63a2cc07cb4a479316 | 127.0.99.135:33161 | HEALTHY | <none> | 2 | 0
694cd43f32a746e7ab0e91db5785ba52 | 127.0.99.130:34403 | HEALTHY | <none> | 3 | 0
9b3cf5210e6b4f2188e66a80dca823e6 | 127.0.99.131:44145 | HEALTHY | <none> | 2 | 0
a9c26918b0724416878324492440c93d | 127.0.99.129:39461 | HEALTHY | <none> | 3 | 0
acc5e87f231a46f5beb89cc24cbe28c4 | 127.0.99.133:45407 | HEALTHY | <none> | 2 | 0
e488cf84f7b74419a218d3512e237fea | 127.0.99.132:40887 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:39461
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:34403
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:44145
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:40887
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:45407
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:38729
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:33161
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:39461
server_dump_info_path | /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:34403
server_dump_info_path | /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:44145
server_dump_info_path | /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:40887
server_dump_info_path | /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:45407
server_dump_info_path | /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:38729
server_dump_info_path | /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:33161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:45197 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 13:58:57.963423 398 log_verifier.cc:126] Checking tablet 03e82ebc742b46ce98fc9ab599b08f46
I20240503 13:58:57.974486 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 03e82ebc742b46ce98fc9ab599b08f46
I20240503 13:58:57.974571 398 log_verifier.cc:126] Checking tablet 0f599173c7624329af379768658f6041
I20240503 13:58:57.983359 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 0f599173c7624329af379768658f6041
I20240503 13:58:57.983415 398 log_verifier.cc:126] Checking tablet 1a0e0ac43355469cbce23fc6a7bb0cfa
I20240503 13:58:57.992080 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1a0e0ac43355469cbce23fc6a7bb0cfa
I20240503 13:58:57.992137 398 log_verifier.cc:126] Checking tablet 1a88836924b04bfea8524184d061a19b
I20240503 13:58:58.001173 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 1a88836924b04bfea8524184d061a19b
I20240503 13:58:58.001235 398 log_verifier.cc:126] Checking tablet 1c36890ea27847caa128b7896559d89f
I20240503 13:58:58.010406 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1c36890ea27847caa128b7896559d89f
I20240503 13:58:58.010474 398 log_verifier.cc:126] Checking tablet 1e949fd674614e29a933cdf304d0ee94
I20240503 13:58:58.019316 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 1e949fd674614e29a933cdf304d0ee94
I20240503 13:58:58.019390 398 log_verifier.cc:126] Checking tablet 25a1559451b8472482ac6e482da41d28
I20240503 13:58:58.028520 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 25a1559451b8472482ac6e482da41d28
I20240503 13:58:58.028596 398 log_verifier.cc:126] Checking tablet 558e844e3d344fcbb0e41cdf132c880c
I20240503 13:58:58.037457 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 558e844e3d344fcbb0e41cdf132c880c
I20240503 13:58:58.037519 398 log_verifier.cc:126] Checking tablet a08c24279c7748dd9b10ff92c6d68845
I20240503 13:58:58.046512 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a08c24279c7748dd9b10ff92c6d68845
I20240503 13:58:58.046583 398 log_verifier.cc:126] Checking tablet c5d7c00e377848949299b739c6ff8d3d
I20240503 13:58:58.055356 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c5d7c00e377848949299b739c6ff8d3d
I20240503 13:58:58.055413 398 log_verifier.cc:126] Checking tablet ce166608b56540d89bddc4919356ebe7
I20240503 13:58:58.064545 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet ce166608b56540d89bddc4919356ebe7
I20240503 13:58:58.064607 398 log_verifier.cc:126] Checking tablet d140fb732825493dbcd7e41ea7e62271
I20240503 13:58:58.073264 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d140fb732825493dbcd7e41ea7e62271
I20240503 13:58:58.073328 398 log_verifier.cc:126] Checking tablet d899ba5a20f0466da4e0e3fc9831e5f5
I20240503 13:58:58.082099 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet d899ba5a20f0466da4e0e3fc9831e5f5
I20240503 13:58:58.082159 398 log_verifier.cc:126] Checking tablet e5e343562d6043b79af234ed9cdd69d4
I20240503 13:58:58.090895 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet e5e343562d6043b79af234ed9cdd69d4
I20240503 13:58:58.090958 398 log_verifier.cc:126] Checking tablet f6f3c78225a94967973739fe8bd5a3c4
I20240503 13:58:58.098696 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet f6f3c78225a94967973739fe8bd5a3c4
I20240503 13:58:58.098757 398 log_verifier.cc:126] Checking tablet fa9ffe2aeedf42e89d4e00ac12fb9e92
I20240503 13:58:58.108004 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet fa9ffe2aeedf42e89d4e00ac12fb9e92
I20240503 13:58:58.109927 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37103
--builtin_ntp_servers=127.0.99.148:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 13:58:58.219723 5797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:58.219969 5797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:58.220031 5797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:58.224226 5797 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:58.224309 5797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:58.224414 5797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 13:58:58.228643 5797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37103
--never_fsync=true
--heap_profile_path=/tmp/kudu.5797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:58.229538 5797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:58.230625 5797 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:58.239339 5802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:58.240162 5803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:58.240909 5805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:58.241799 5797 server_base.cc:1034] running on GCE node
I20240503 13:58:58.242174 5797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:58.242693 5797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:58.243835 5797 hybrid_clock.cc:648] HybridClock initialized: now 1714744738243817 us; error 35 us; skew 500 ppm
I20240503 13:58:58.245618 5797 webserver.cc:460] Webserver started at http://127.0.99.136:39269/ using document root <none> and password file <none>
I20240503 13:58:58.246124 5797 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:58.246179 5797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:58.246343 5797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:58.247942 5797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "5beaa5ed536c4caf8fd963010359595b"
format_stamp: "Formatted at 2024-05-03 13:58:58 on dist-test-slave-fr9h"
I20240503 13:58:58.248378 5797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "5beaa5ed536c4caf8fd963010359595b"
format_stamp: "Formatted at 2024-05-03 13:58:58 on dist-test-slave-fr9h"
I20240503 13:58:58.251407 5797 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 13:58:58.253405 5811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:58.254328 5797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:58.254434 5797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "5beaa5ed536c4caf8fd963010359595b"
format_stamp: "Formatted at 2024-05-03 13:58:58 on dist-test-slave-fr9h"
I20240503 13:58:58.254520 5797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:58.274309 5797 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:58.275377 5797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:58.275523 5797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:58.275992 5797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:58.276770 5797 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:58.276816 5797 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:58.276868 5797 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:58.276893 5797 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:58.284958 5797 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:43465
I20240503 13:58:58.284991 5924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:43465 every 8 connection(s)
I20240503 13:58:58.285812 5797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 13:58:58.294267 5925 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37103
I20240503 13:58:58.294478 5925 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:58.294934 5925 heartbeater.cc:511] Master 127.0.99.190:37103 requested a full tablet report, sending...
I20240503 13:58:58.295635 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 5797
I20240503 13:58:58.295744 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 13:58:58.295956 2789 ts_manager.cc:194] Registered new tserver with Master: 5beaa5ed536c4caf8fd963010359595b (127.0.99.136:43465)
I20240503 13:58:58.296677 2789 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:53933
I20240503 13:58:58.482164 4561 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:58.482604 5859 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:58.483099 4430 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:58.484736 3042 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:58.492183 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:58.495864 4168 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:58.511775 2912 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:58.517204 4299 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:58.620536 2912 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:58.621320 4430 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:58.622961 5859 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:58.629527 4299 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:58.636094 3042 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:58.636919 4168 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:58.647085 4561 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:58.657776 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:58.684194 3064 consensus_queue.cc:237] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: true } }
I20240503 13:58:58.685142 6018 raft_consensus.cc:2932] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5beaa5ed536c4caf8fd963010359595b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: true } } }
I20240503 13:58:58.688992 2788 catalog_manager.cc:5484] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 reported cstate change: config changed from index -1 to 4, NON_VOTER 5beaa5ed536c4caf8fd963010359595b (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "694cd43f32a746e7ab0e91db5785ba52" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 13:58:58.689357 2998 consensus_peers.cc:485] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 -> Peer 5beaa5ed536c4caf8fd963010359595b (127.0.99.136:43465): Couldn't send request to peer 5beaa5ed536c4caf8fd963010359595b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d140fb732825493dbcd7e41ea7e62271. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:58.694823 2929 consensus_queue.cc:237] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: true } }
I20240503 13:58:58.695727 6031 raft_consensus.cc:2932] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5beaa5ed536c4caf8fd963010359595b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: true } } }
I20240503 13:58:58.698050 2788 catalog_manager.cc:5484] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d reported cstate change: config changed from index -1 to 4, NON_VOTER 5beaa5ed536c4caf8fd963010359595b (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "a9c26918b0724416878324492440c93d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 13:58:58.699493 2867 consensus_peers.cc:485] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d -> Peer 5beaa5ed536c4caf8fd963010359595b (127.0.99.136:43465): Couldn't send request to peer 5beaa5ed536c4caf8fd963010359595b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5d7c00e377848949299b739c6ff8d3d. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:59.127419 6056 ts_tablet_manager.cc:902] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b: Initiating tablet copy from peer 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130:34403)
I20240503 13:58:59.128276 6056 tablet_copy_client.cc:308] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34403
I20240503 13:58:59.131986 3084 tablet_copy_service.cc:140] P 694cd43f32a746e7ab0e91db5785ba52: Received BeginTabletCopySession request for tablet d140fb732825493dbcd7e41ea7e62271 from peer 5beaa5ed536c4caf8fd963010359595b ({username='slave'} at 127.0.99.136:52399)
I20240503 13:58:59.132102 3084 tablet_copy_service.cc:161] P 694cd43f32a746e7ab0e91db5785ba52: Beginning new tablet copy session on tablet d140fb732825493dbcd7e41ea7e62271 from peer 5beaa5ed536c4caf8fd963010359595b at {username='slave'} at 127.0.99.136:52399: session id = 5beaa5ed536c4caf8fd963010359595b-d140fb732825493dbcd7e41ea7e62271
I20240503 13:58:59.132889 3084 tablet_copy_source_session.cc:205] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:58:59.136319 6056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d140fb732825493dbcd7e41ea7e62271. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:59.140892 6056 tablet_copy_client.cc:783] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b: tablet copy: Starting download of 0 data blocks...
I20240503 13:58:59.141095 6056 tablet_copy_client.cc:647] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b: tablet copy: Starting download of 1 WAL segments...
I20240503 13:58:59.142438 6056 tablet_copy_client.cc:523] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:58:59.143849 6056 tablet_bootstrap.cc:492] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b: Bootstrap starting.
I20240503 13:58:59.148602 6056 log.cc:824] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:59.166525 6056 tablet_bootstrap.cc:492] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:58:59.166798 6056 tablet_bootstrap.cc:492] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b: Bootstrap complete.
I20240503 13:58:59.167038 6056 ts_tablet_manager.cc:1367] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b: Time spent bootstrapping tablet: real 0.023s user 0.020s sys 0.000s
I20240503 13:58:59.168884 6056 raft_consensus.cc:348] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: true } }
I20240503 13:58:59.169067 6056 raft_consensus.cc:729] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5beaa5ed536c4caf8fd963010359595b, State: Initialized, Role: LEARNER
I20240503 13:58:59.169354 6056 consensus_queue.cc:260] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: true } }
I20240503 13:58:59.169940 5925 heartbeater.cc:503] Master 127.0.99.190:37103 was elected leader, sending a full tablet report...
I20240503 13:58:59.170047 6056 ts_tablet_manager.cc:1398] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20240503 13:58:59.170765 3084 tablet_copy_service.cc:342] P 694cd43f32a746e7ab0e91db5785ba52: Request end of tablet copy session 5beaa5ed536c4caf8fd963010359595b-d140fb732825493dbcd7e41ea7e62271 received from {username='slave'} at 127.0.99.136:52399
I20240503 13:58:59.170855 3084 tablet_copy_service.cc:433] P 694cd43f32a746e7ab0e91db5785ba52: ending tablet copy session 5beaa5ed536c4caf8fd963010359595b-d140fb732825493dbcd7e41ea7e62271 on tablet d140fb732825493dbcd7e41ea7e62271 with peer 5beaa5ed536c4caf8fd963010359595b
I20240503 13:58:59.191360 6056 ts_tablet_manager.cc:902] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b: Initiating tablet copy from peer a9c26918b0724416878324492440c93d (127.0.99.129:39461)
I20240503 13:58:59.191767 6056 tablet_copy_client.cc:308] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:39461
I20240503 13:58:59.195010 2952 tablet_copy_service.cc:140] P a9c26918b0724416878324492440c93d: Received BeginTabletCopySession request for tablet c5d7c00e377848949299b739c6ff8d3d from peer 5beaa5ed536c4caf8fd963010359595b ({username='slave'} at 127.0.99.136:42731)
I20240503 13:58:59.195129 2952 tablet_copy_service.cc:161] P a9c26918b0724416878324492440c93d: Beginning new tablet copy session on tablet c5d7c00e377848949299b739c6ff8d3d from peer 5beaa5ed536c4caf8fd963010359595b at {username='slave'} at 127.0.99.136:42731: session id = 5beaa5ed536c4caf8fd963010359595b-c5d7c00e377848949299b739c6ff8d3d
I20240503 13:58:59.195909 2952 tablet_copy_source_session.cc:205] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:58:59.196707 6056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5d7c00e377848949299b739c6ff8d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:59.198472 6056 tablet_copy_client.cc:783] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b: tablet copy: Starting download of 0 data blocks...
I20240503 13:58:59.198607 6056 tablet_copy_client.cc:647] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b: tablet copy: Starting download of 1 WAL segments...
I20240503 13:58:59.199784 6056 tablet_copy_client.cc:523] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:58:59.201768 6056 tablet_bootstrap.cc:492] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b: Bootstrap starting.
I20240503 13:58:59.219179 6056 tablet_bootstrap.cc:492] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:58:59.219485 6056 tablet_bootstrap.cc:492] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b: Bootstrap complete.
I20240503 13:58:59.219657 6056 ts_tablet_manager.cc:1367] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.003s
I20240503 13:58:59.220183 6056 raft_consensus.cc:348] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: true } }
I20240503 13:58:59.220340 6056 raft_consensus.cc:729] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5beaa5ed536c4caf8fd963010359595b, State: Initialized, Role: LEARNER
I20240503 13:58:59.220438 6056 consensus_queue.cc:260] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: true } }
I20240503 13:58:59.220671 6056 ts_tablet_manager.cc:1398] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:59.221230 2952 tablet_copy_service.cc:342] P a9c26918b0724416878324492440c93d: Request end of tablet copy session 5beaa5ed536c4caf8fd963010359595b-c5d7c00e377848949299b739c6ff8d3d received from {username='slave'} at 127.0.99.136:42731
I20240503 13:58:59.221328 2952 tablet_copy_service.cc:433] P a9c26918b0724416878324492440c93d: ending tablet copy session 5beaa5ed536c4caf8fd963010359595b-c5d7c00e377848949299b739c6ff8d3d on tablet c5d7c00e377848949299b739c6ff8d3d with peer 5beaa5ed536c4caf8fd963010359595b
I20240503 13:58:59.548802 5879 raft_consensus.cc:1203] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:58:59.670097 5879 raft_consensus.cc:1203] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:00.108477 6122 raft_consensus.cc:1050] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52: attempting to promote NON_VOTER 5beaa5ed536c4caf8fd963010359595b to VOTER
I20240503 13:59:00.108853 6122 consensus_queue.cc:237] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: false } }
I20240503 13:59:00.109911 5879 raft_consensus.cc:1261] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b [term 1 LEARNER]: Refusing update from remote peer 694cd43f32a746e7ab0e91db5785ba52: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:00.110347 6122 consensus_queue.cc:1035] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:00.112845 6121 raft_consensus.cc:2932] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5beaa5ed536c4caf8fd963010359595b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: false } } }
I20240503 13:59:00.113386 5879 raft_consensus.cc:2932] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5beaa5ed536c4caf8fd963010359595b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: false } } }
I20240503 13:59:00.115464 2788 catalog_manager.cc:5484] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 reported cstate change: config changed from index 4 to 5, 5beaa5ed536c4caf8fd963010359595b (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "694cd43f32a746e7ab0e91db5785ba52" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:00.130959 6100 raft_consensus.cc:1050] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d: attempting to promote NON_VOTER 5beaa5ed536c4caf8fd963010359595b to VOTER
I20240503 13:59:00.131266 6100 consensus_queue.cc:237] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: false } }
I20240503 13:59:00.132308 5879 raft_consensus.cc:1261] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b [term 1 LEARNER]: Refusing update from remote peer a9c26918b0724416878324492440c93d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:00.132654 6100 consensus_queue.cc:1035] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:00.134245 6033 raft_consensus.cc:2932] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5beaa5ed536c4caf8fd963010359595b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: false } } }
I20240503 13:59:00.134665 5879 raft_consensus.cc:2932] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5beaa5ed536c4caf8fd963010359595b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: false } } }
I20240503 13:59:00.136847 2790 catalog_manager.cc:5484] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d reported cstate change: config changed from index 4 to 5, 5beaa5ed536c4caf8fd963010359595b (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a9c26918b0724416878324492440c93d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:00.196597 2929 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c5d7c00e377848949299b739c6ff8d3d"
dest_uuid: "a9c26918b0724416878324492440c93d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43402
I20240503 13:59:00.196731 2929 raft_consensus.cc:595] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:00.206086 3064 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d140fb732825493dbcd7e41ea7e62271"
dest_uuid: "694cd43f32a746e7ab0e91db5785ba52"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40312
I20240503 13:59:00.206213 3064 raft_consensus.cc:595] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:00.416500 2929 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c5d7c00e377848949299b739c6ff8d3d"
dest_uuid: "a9c26918b0724416878324492440c93d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43422
I20240503 13:59:00.416627 2929 raft_consensus.cc:595] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:00.426146 3064 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d140fb732825493dbcd7e41ea7e62271"
dest_uuid: "694cd43f32a746e7ab0e91db5785ba52"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40334
I20240503 13:59:00.426283 3064 raft_consensus.cc:595] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:00.620530 6099 raft_consensus.cc:979] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d: : Instructing follower 5beaa5ed536c4caf8fd963010359595b to start an election
I20240503 13:59:00.620644 6099 raft_consensus.cc:1067] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [term 1 LEADER]: Signalling peer 5beaa5ed536c4caf8fd963010359595b to start an election
I20240503 13:59:00.621091 5879 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c5d7c00e377848949299b739c6ff8d3d"
dest_uuid: "5beaa5ed536c4caf8fd963010359595b"
from {username='slave'} at 127.0.99.129:47701
I20240503 13:59:00.621280 5879 raft_consensus.cc:482] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:00.621394 5879 raft_consensus.cc:3037] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:00.622344 5879 raft_consensus.cc:504] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: false } }
I20240503 13:59:00.623303 5879 leader_election.cc:288] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b [CANDIDATE]: Term 2 election: Requested vote from peers a9c26918b0724416878324492440c93d (127.0.99.129:39461)
I20240503 13:59:00.623822 2929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d7c00e377848949299b739c6ff8d3d" candidate_uuid: "5beaa5ed536c4caf8fd963010359595b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a9c26918b0724416878324492440c93d"
I20240503 13:59:00.623931 2929 raft_consensus.cc:3032] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:00.623977 2929 raft_consensus.cc:729] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a9c26918b0724416878324492440c93d, State: Running, Role: LEADER
I20240503 13:59:00.624112 2929 consensus_queue.cc:260] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: false } }
I20240503 13:59:00.624235 2929 raft_consensus.cc:3037] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:00.625046 2929 raft_consensus.cc:2454] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beaa5ed536c4caf8fd963010359595b in term 2.
I20240503 13:59:00.625411 5812 leader_election.cc:302] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5beaa5ed536c4caf8fd963010359595b, a9c26918b0724416878324492440c93d; no voters:
I20240503 13:59:00.625636 6071 raft_consensus.cc:2781] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:00.625844 6071 raft_consensus.cc:686] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b [term 2 LEADER]: Becoming Leader. State: Replica: 5beaa5ed536c4caf8fd963010359595b, State: Running, Role: LEADER
I20240503 13:59:00.626120 6071 consensus_queue.cc:237] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } } peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: false } }
I20240503 13:59:00.628187 2790 catalog_manager.cc:5484] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b reported cstate change: term changed from 1 to 2, leader changed from a9c26918b0724416878324492440c93d (127.0.99.129) to 5beaa5ed536c4caf8fd963010359595b (127.0.99.136). New cstate: current_term: 2 leader_uuid: "5beaa5ed536c4caf8fd963010359595b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 13:59:00.634323 2776 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet c5d7c00e377848949299b739c6ff8d3d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 5beaa5ed536c4caf8fd963010359595b (127.0.99.136:43465) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240503 13:59:00.642757 3064 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d140fb732825493dbcd7e41ea7e62271"
dest_uuid: "694cd43f32a746e7ab0e91db5785ba52"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40348
I20240503 13:59:00.642897 3064 raft_consensus.cc:595] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:00.732831 6190 raft_consensus.cc:979] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52: : Instructing follower 5beaa5ed536c4caf8fd963010359595b to start an election
I20240503 13:59:00.732970 6190 raft_consensus.cc:1067] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Signalling peer 5beaa5ed536c4caf8fd963010359595b to start an election
I20240503 13:59:00.733409 5879 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d140fb732825493dbcd7e41ea7e62271"
dest_uuid: "5beaa5ed536c4caf8fd963010359595b"
from {username='slave'} at 127.0.99.130:44537
I20240503 13:59:00.733556 5879 raft_consensus.cc:482] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:00.733644 5879 raft_consensus.cc:3037] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:00.734648 5879 raft_consensus.cc:504] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: false } }
I20240503 13:59:00.735033 5879 leader_election.cc:288] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b [CANDIDATE]: Term 2 election: Requested vote from peers 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130:34403)
I20240503 13:59:00.735466 3064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d140fb732825493dbcd7e41ea7e62271" candidate_uuid: "5beaa5ed536c4caf8fd963010359595b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "694cd43f32a746e7ab0e91db5785ba52"
I20240503 13:59:00.735567 3064 raft_consensus.cc:3032] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:00.735603 3064 raft_consensus.cc:729] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 694cd43f32a746e7ab0e91db5785ba52, State: Running, Role: LEADER
I20240503 13:59:00.735742 3064 consensus_queue.cc:260] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: false } }
I20240503 13:59:00.735863 3064 raft_consensus.cc:3037] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:00.736552 3064 raft_consensus.cc:2454] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beaa5ed536c4caf8fd963010359595b in term 2.
I20240503 13:59:00.736851 5813 leader_election.cc:302] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5beaa5ed536c4caf8fd963010359595b, 694cd43f32a746e7ab0e91db5785ba52; no voters:
I20240503 13:59:00.737030 6071 raft_consensus.cc:2781] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:00.737133 6071 raft_consensus.cc:686] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b [term 2 LEADER]: Becoming Leader. State: Replica: 5beaa5ed536c4caf8fd963010359595b, State: Running, Role: LEADER
I20240503 13:59:00.737267 6071 consensus_queue.cc:237] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } } peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: false } }
I20240503 13:59:00.738602 2790 catalog_manager.cc:5484] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b reported cstate change: term changed from 1 to 2, leader changed from 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130) to 5beaa5ed536c4caf8fd963010359595b (127.0.99.136). New cstate: current_term: 2 leader_uuid: "5beaa5ed536c4caf8fd963010359595b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:01.019501 2929 raft_consensus.cc:1261] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [term 2 FOLLOWER]: Refusing update from remote peer 5beaa5ed536c4caf8fd963010359595b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 13:59:01.020040 6071 consensus_queue.cc:1035] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9c26918b0724416878324492440c93d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:01.247320 3064 raft_consensus.cc:1261] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [term 2 FOLLOWER]: Refusing update from remote peer 5beaa5ed536c4caf8fd963010359595b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 13:59:01.247854 6071 consensus_queue.cc:1035] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b [LEADER]: Connected to new peer: Peer: permanent_uuid: "694cd43f32a746e7ab0e91db5785ba52" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:01.253336 5879 consensus_queue.cc:237] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: false } }
I20240503 13:59:01.253841 6071 raft_consensus.cc:2932] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER a9c26918b0724416878324492440c93d (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: false } } }
I20240503 13:59:01.255038 2776 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c5d7c00e377848949299b739c6ff8d3d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 13:59:01.255702 2790 catalog_manager.cc:5484] T c5d7c00e377848949299b739c6ff8d3d P 5beaa5ed536c4caf8fd963010359595b reported cstate change: config changed from index 5 to 7, VOTER a9c26918b0724416878324492440c93d (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "5beaa5ed536c4caf8fd963010359595b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:01.258356 2912 tablet_service.cc:1514] Processing DeleteTablet for tablet c5d7c00e377848949299b739c6ff8d3d with delete_type TABLET_DATA_TOMBSTONED (TS a9c26918b0724416878324492440c93d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46422
I20240503 13:59:01.258719 6220 tablet_replica.cc:330] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d: stopping tablet replica
I20240503 13:59:01.258834 6220 raft_consensus.cc:2229] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:01.259073 6220 raft_consensus.cc:2258] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:01.260419 6220 ts_tablet_manager.cc:1875] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:01.262275 6220 ts_tablet_manager.cc:1888] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 13:59:01.262374 6220 log.cc:1196] T c5d7c00e377848949299b739c6ff8d3d P a9c26918b0724416878324492440c93d: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/c5d7c00e377848949299b739c6ff8d3d
I20240503 13:59:01.262878 2775 catalog_manager.cc:4830] TS a9c26918b0724416878324492440c93d (127.0.99.129:39461): tablet c5d7c00e377848949299b739c6ff8d3d (table rebalance_test_table_4 [id=3860ccce8f8f4a17b1dc250f91397df8]) successfully deleted
I20240503 13:59:01.367643 5879 consensus_queue.cc:237] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: false } }
I20240503 13:59:01.368408 6071 raft_consensus.cc:2932] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: false } } }
I20240503 13:59:01.369668 2776 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d140fb732825493dbcd7e41ea7e62271 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 13:59:01.370533 2790 catalog_manager.cc:5484] T d140fb732825493dbcd7e41ea7e62271 P 5beaa5ed536c4caf8fd963010359595b reported cstate change: config changed from index 5 to 7, VOTER 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "5beaa5ed536c4caf8fd963010359595b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5beaa5ed536c4caf8fd963010359595b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43465 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:01.372983 3042 tablet_service.cc:1514] Processing DeleteTablet for tablet d140fb732825493dbcd7e41ea7e62271 with delete_type TABLET_DATA_TOMBSTONED (TS 694cd43f32a746e7ab0e91db5785ba52 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49062
I20240503 13:59:01.373389 6236 tablet_replica.cc:330] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52: stopping tablet replica
I20240503 13:59:01.373533 6236 raft_consensus.cc:2229] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:01.373657 6236 raft_consensus.cc:2258] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:01.374924 6236 ts_tablet_manager.cc:1875] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:01.376719 6236 ts_tablet_manager.cc:1888] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 13:59:01.376785 6236 log.cc:1196] T d140fb732825493dbcd7e41ea7e62271 P 694cd43f32a746e7ab0e91db5785ba52: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/d140fb732825493dbcd7e41ea7e62271
I20240503 13:59:01.377292 2776 catalog_manager.cc:4830] TS 694cd43f32a746e7ab0e91db5785ba52 (127.0.99.130:34403): tablet d140fb732825493dbcd7e41ea7e62271 (table rebalance_test_table_3 [id=785472d7e43343c0a92a6afb07b14c83]) successfully deleted
I20240503 13:59:01.549067 5859 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:01.549191 4430 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:01.551349 4561 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:01.561218 3042 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:01.565418 2912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:01.567488 4299 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:01.574470 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:01.581395 4168 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:01.672322 3042 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:01.674131 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:01.679495 4168 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:01.681180 4430 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:01.684141 5859 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:01.692099 4561 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:01.692602 2912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:01.699636 4299 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:01.801864 4430 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:01.803594 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:01.807488 2912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:01.809216 4561 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:01.817427 4299 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:01.821974 4168 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:01.829365 5859 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:01.834971 3042 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b0ba17e1ed7e481fa58f7ce6b035ccaa | 127.0.99.190:37103 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:45197 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
3b1cfcd5138346b6ac5b8581c536034d | 127.0.99.134:38729 | HEALTHY | <none> | 2 | 0
5beaa5ed536c4caf8fd963010359595b | 127.0.99.136:43465 | HEALTHY | <none> | 2 | 0
5c33992e399d4e63a2cc07cb4a479316 | 127.0.99.135:33161 | HEALTHY | <none> | 2 | 0
694cd43f32a746e7ab0e91db5785ba52 | 127.0.99.130:34403 | HEALTHY | <none> | 2 | 0
9b3cf5210e6b4f2188e66a80dca823e6 | 127.0.99.131:44145 | HEALTHY | <none> | 2 | 0
a9c26918b0724416878324492440c93d | 127.0.99.129:39461 | HEALTHY | <none> | 2 | 0
acc5e87f231a46f5beb89cc24cbe28c4 | 127.0.99.133:45407 | HEALTHY | <none> | 2 | 0
e488cf84f7b74419a218d3512e237fea | 127.0.99.132:40887 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:39461
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:34403
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:44145
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:40887
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:45407
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:38729
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:33161
local_ip_for_outbound_sockets | 127.0.99.136 | experimental | 127.0.99.136:43465
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:39461
server_dump_info_path | /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:34403
server_dump_info_path | /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:44145
server_dump_info_path | /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:40887
server_dump_info_path | /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:45407
server_dump_info_path | /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:38729
server_dump_info_path | /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:33161
server_dump_info_path | /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.0.99.136:43465
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:45197 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 13:59:01.860743 398 log_verifier.cc:126] Checking tablet 03e82ebc742b46ce98fc9ab599b08f46
I20240503 13:59:01.868146 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 03e82ebc742b46ce98fc9ab599b08f46
I20240503 13:59:01.868208 398 log_verifier.cc:126] Checking tablet 0f599173c7624329af379768658f6041
I20240503 13:59:01.874104 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 0f599173c7624329af379768658f6041
I20240503 13:59:01.874158 398 log_verifier.cc:126] Checking tablet 1a0e0ac43355469cbce23fc6a7bb0cfa
I20240503 13:59:01.879822 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1a0e0ac43355469cbce23fc6a7bb0cfa
I20240503 13:59:01.879886 398 log_verifier.cc:126] Checking tablet 1a88836924b04bfea8524184d061a19b
I20240503 13:59:01.885628 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 1a88836924b04bfea8524184d061a19b
I20240503 13:59:01.885684 398 log_verifier.cc:126] Checking tablet 1c36890ea27847caa128b7896559d89f
I20240503 13:59:01.891006 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1c36890ea27847caa128b7896559d89f
I20240503 13:59:01.891059 398 log_verifier.cc:126] Checking tablet 1e949fd674614e29a933cdf304d0ee94
I20240503 13:59:01.896739 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 1e949fd674614e29a933cdf304d0ee94
I20240503 13:59:01.896795 398 log_verifier.cc:126] Checking tablet 25a1559451b8472482ac6e482da41d28
I20240503 13:59:01.902724 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 25a1559451b8472482ac6e482da41d28
I20240503 13:59:01.902789 398 log_verifier.cc:126] Checking tablet 558e844e3d344fcbb0e41cdf132c880c
I20240503 13:59:01.908977 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 558e844e3d344fcbb0e41cdf132c880c
I20240503 13:59:01.909034 398 log_verifier.cc:126] Checking tablet a08c24279c7748dd9b10ff92c6d68845
I20240503 13:59:01.914523 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a08c24279c7748dd9b10ff92c6d68845
I20240503 13:59:01.914582 398 log_verifier.cc:126] Checking tablet c5d7c00e377848949299b739c6ff8d3d
I20240503 13:59:01.922878 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c5d7c00e377848949299b739c6ff8d3d
I20240503 13:59:01.922940 398 log_verifier.cc:126] Checking tablet ce166608b56540d89bddc4919356ebe7
I20240503 13:59:01.929239 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet ce166608b56540d89bddc4919356ebe7
I20240503 13:59:01.929296 398 log_verifier.cc:126] Checking tablet d140fb732825493dbcd7e41ea7e62271
I20240503 13:59:01.937429 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet d140fb732825493dbcd7e41ea7e62271
I20240503 13:59:01.937489 398 log_verifier.cc:126] Checking tablet d899ba5a20f0466da4e0e3fc9831e5f5
I20240503 13:59:01.943153 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet d899ba5a20f0466da4e0e3fc9831e5f5
I20240503 13:59:01.943207 398 log_verifier.cc:126] Checking tablet e5e343562d6043b79af234ed9cdd69d4
I20240503 13:59:01.948726 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet e5e343562d6043b79af234ed9cdd69d4
I20240503 13:59:01.948773 398 log_verifier.cc:126] Checking tablet f6f3c78225a94967973739fe8bd5a3c4
I20240503 13:59:01.953140 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet f6f3c78225a94967973739fe8bd5a3c4
I20240503 13:59:01.953187 398 log_verifier.cc:126] Checking tablet fa9ffe2aeedf42e89d4e00ac12fb9e92
I20240503 13:59:01.958881 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet fa9ffe2aeedf42e89d4e00ac12fb9e92
I20240503 13:59:01.959350 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 2851
I20240503 13:59:01.966048 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 2982
I20240503 13:59:01.972258 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 3975
I20240503 13:59:01.978266 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 4106
I20240503 13:59:01.984153 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 4237
I20240503 13:59:01.990422 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 4367
I20240503 13:59:01.996248 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 4499
I20240503 13:59:02.001670 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 5797
I20240503 13:59:02.007490 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 2760
2024-05-03T13:59:02Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/1 (18022 ms)
[----------] 1 test from RebalanceParamTest (18022 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1
I20240503 13:59:02.047637 398 test_util.cc:268] Using random seed: 523956109
I20240503 13:59:02.048522 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 13:59:02.048610 398 ts_itest-base.cc:113] --------------
I20240503 13:59:02.048671 398 ts_itest-base.cc:114] 8 tablet servers
I20240503 13:59:02.048703 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 13:59:02.048736 398 ts_itest-base.cc:116] --------------
2024-05-03T13:59:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T13:59:02Z Disabled control of system clock
I20240503 13:59:02.057432 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:45485
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:45485
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 13:59:02.163057 6356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:02.163265 6356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:02.163339 6356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:02.166316 6356 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:02.166378 6356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 13:59:02.166401 6356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:02.166424 6356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 13:59:02.166471 6356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 13:59:02.169970 6356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:45485
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:45485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.6356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:02.170800 6356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:02.171940 6356 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:02.179634 6362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:02.180409 6361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:02.181154 6364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:02.181985 6356 server_base.cc:1034] running on GCE node
I20240503 13:59:02.182510 6356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:02.183410 6356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:02.184590 6356 hybrid_clock.cc:648] HybridClock initialized: now 1714744742184568 us; error 36 us; skew 500 ppm
I20240503 13:59:02.186367 6356 webserver.cc:460] Webserver started at http://127.0.99.190:36663/ using document root <none> and password file <none>
I20240503 13:59:02.186923 6356 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:02.186983 6356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:02.187163 6356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:02.188820 6356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "453655c40fe0450b838a8fa099757ca4"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-fr9h"
I20240503 13:59:02.189280 6356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "453655c40fe0450b838a8fa099757ca4"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-fr9h"
I20240503 13:59:02.192503 6356 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240503 13:59:02.194571 6370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.195541 6356 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:02.195657 6356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "453655c40fe0450b838a8fa099757ca4"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-fr9h"
I20240503 13:59:02.195735 6356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:02.210899 6356 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:02.211524 6356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:02.211678 6356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:02.218447 6356 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:45485
I20240503 13:59:02.218508 6422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:45485 every 8 connection(s)
I20240503 13:59:02.219460 6356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 13:59:02.222031 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 6356
I20240503 13:59:02.222146 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 13:59:02.222290 6423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:02.228258 6423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Bootstrap starting.
I20240503 13:59:02.230525 6423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:02.231186 6423 log.cc:824] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:02.232800 6423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: No bootstrap required, opened a new log
I20240503 13:59:02.235199 6423 raft_consensus.cc:348] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453655c40fe0450b838a8fa099757ca4" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 45485 } }
I20240503 13:59:02.235428 6423 raft_consensus.cc:374] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:02.235467 6423 raft_consensus.cc:729] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 453655c40fe0450b838a8fa099757ca4, State: Initialized, Role: FOLLOWER
I20240503 13:59:02.235867 6423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453655c40fe0450b838a8fa099757ca4" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 45485 } }
I20240503 13:59:02.235981 6423 raft_consensus.cc:388] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:02.236027 6423 raft_consensus.cc:482] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:02.236112 6423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:02.236797 6423 raft_consensus.cc:504] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453655c40fe0450b838a8fa099757ca4" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 45485 } }
I20240503 13:59:02.237116 6423 leader_election.cc:302] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 453655c40fe0450b838a8fa099757ca4; no voters:
I20240503 13:59:02.237408 6423 leader_election.cc:288] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:02.237780 6428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:02.237989 6428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4 [term 1 LEADER]: Becoming Leader. State: Replica: 453655c40fe0450b838a8fa099757ca4, State: Running, Role: LEADER
I20240503 13:59:02.238255 6428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453655c40fe0450b838a8fa099757ca4" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 45485 } }
I20240503 13:59:02.238483 6423 sys_catalog.cc:564] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 13:59:02.239624 6429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "453655c40fe0450b838a8fa099757ca4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453655c40fe0450b838a8fa099757ca4" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 45485 } } }
I20240503 13:59:02.239735 6429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4 [sys.catalog]: This master's current role is: LEADER
I20240503 13:59:02.240031 6430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 453655c40fe0450b838a8fa099757ca4. Latest consensus state: current_term: 1 leader_uuid: "453655c40fe0450b838a8fa099757ca4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453655c40fe0450b838a8fa099757ca4" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 45485 } } }
I20240503 13:59:02.240114 6430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4 [sys.catalog]: This master's current role is: LEADER
I20240503 13:59:02.240200 6437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 13:59:02.242776 6437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 13:59:02.247407 6437 catalog_manager.cc:1347] Generated new cluster ID: ca3c65c63e0842b7aa61e46c39b80a3b
I20240503 13:59:02.247480 6437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 13:59:02.261042 6437 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 13:59:02.262053 6437 catalog_manager.cc:1504] Loading token signing keys...
I20240503 13:59:02.267413 6437 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Generated new TSK 0
I20240503 13:59:02.268127 6437 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 13:59:02.273355 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45485
--builtin_ntp_servers=127.0.99.148:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 13:59:02.377069 6447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:02.377264 6447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:02.377305 6447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:02.380312 6447 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:02.380375 6447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:02.380458 6447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 13:59:02.384140 6447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45485
--never_fsync=true
--heap_profile_path=/tmp/kudu.6447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:02.384980 6447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:02.386054 6447 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:02.394831 6452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:02.395581 6453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:02.396364 6455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:02.397289 6447 server_base.cc:1034] running on GCE node
I20240503 13:59:02.397703 6447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:02.398336 6447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:02.399497 6447 hybrid_clock.cc:648] HybridClock initialized: now 1714744742399479 us; error 29 us; skew 500 ppm
I20240503 13:59:02.401715 6447 webserver.cc:460] Webserver started at http://127.0.99.129:38965/ using document root <none> and password file <none>
I20240503 13:59:02.402240 6447 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:02.402290 6447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:02.402462 6447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:02.404089 6447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "5f46e19644fd4f4b825c86ba6af768da"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-fr9h"
I20240503 13:59:02.404518 6447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "5f46e19644fd4f4b825c86ba6af768da"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-fr9h"
I20240503 13:59:02.407605 6447 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 13:59:02.409600 6461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.410867 6447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:02.410982 6447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "5f46e19644fd4f4b825c86ba6af768da"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-fr9h"
I20240503 13:59:02.411075 6447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:02.437685 6447 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:02.438324 6447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:02.438522 6447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:02.439090 6447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:02.440043 6447 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:02.440095 6447 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.440153 6447 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:02.440183 6447 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.448460 6447 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:38021
I20240503 13:59:02.448498 6574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:38021 every 8 connection(s)
I20240503 13:59:02.449483 6447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 13:59:02.457845 6575 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45485
I20240503 13:59:02.458051 6575 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:02.458513 6575 heartbeater.cc:511] Master 127.0.99.190:45485 requested a full tablet report, sending...
I20240503 13:59:02.459209 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 6447
I20240503 13:59:02.459357 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 13:59:02.460026 6387 ts_manager.cc:194] Registered new tserver with Master: 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:02.461376 6387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:56079
I20240503 13:59:02.461668 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45485
--builtin_ntp_servers=127.0.99.148:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 13:59:02.563789 6578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:02.564021 6578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:02.564082 6578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:02.567366 6578 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:02.567448 6578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:02.567567 6578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 13:59:02.572530 6578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45485
--never_fsync=true
--heap_profile_path=/tmp/kudu.6578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:02.573630 6578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:02.574972 6578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:02.583806 6586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:02.584547 6583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:02.585366 6584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:02.586169 6578 server_base.cc:1034] running on GCE node
I20240503 13:59:02.586534 6578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:02.587095 6578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:02.588251 6578 hybrid_clock.cc:648] HybridClock initialized: now 1714744742588236 us; error 33 us; skew 500 ppm
I20240503 13:59:02.590061 6578 webserver.cc:460] Webserver started at http://127.0.99.130:44733/ using document root <none> and password file <none>
I20240503 13:59:02.590620 6578 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:02.590677 6578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:02.590845 6578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:02.592458 6578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "2145d15ab2d74c0287839266f1411a43"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-fr9h"
I20240503 13:59:02.592941 6578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "2145d15ab2d74c0287839266f1411a43"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-fr9h"
I20240503 13:59:02.596120 6578 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 13:59:02.598220 6592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.599223 6578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:02.599346 6578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "2145d15ab2d74c0287839266f1411a43"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-fr9h"
I20240503 13:59:02.599431 6578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:02.616948 6578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:02.617525 6578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:02.617667 6578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:02.618139 6578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:02.618974 6578 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:02.619022 6578 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.619061 6578 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:02.619082 6578 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.627067 6578 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:46119
I20240503 13:59:02.627120 6705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:46119 every 8 connection(s)
I20240503 13:59:02.628046 6578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 13:59:02.636217 6706 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45485
I20240503 13:59:02.636430 6706 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:02.636934 6706 heartbeater.cc:511] Master 127.0.99.190:45485 requested a full tablet report, sending...
I20240503 13:59:02.637034 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 6578
I20240503 13:59:02.637159 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 13:59:02.637907 6387 ts_manager.cc:194] Registered new tserver with Master: 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:02.638496 6387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:40749
I20240503 13:59:02.639242 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45485
--builtin_ntp_servers=127.0.99.148:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 13:59:02.741628 6709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:02.741837 6709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:02.741885 6709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:02.745076 6709 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:02.745141 6709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:02.745239 6709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 13:59:02.748863 6709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45485
--never_fsync=true
--heap_profile_path=/tmp/kudu.6709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:02.749744 6709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:02.750831 6709 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:02.759492 6715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:02.760247 6717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:02.761022 6714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:02.761862 6709 server_base.cc:1034] running on GCE node
I20240503 13:59:02.762231 6709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:02.762809 6709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:02.763964 6709 hybrid_clock.cc:648] HybridClock initialized: now 1714744742763946 us; error 30 us; skew 500 ppm
I20240503 13:59:02.765779 6709 webserver.cc:460] Webserver started at http://127.0.99.131:46605/ using document root <none> and password file <none>
I20240503 13:59:02.766336 6709 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:02.766397 6709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:02.766574 6709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:02.768189 6709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "4a85e492c8154e9881a9cc2a105d7e64"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-fr9h"
I20240503 13:59:02.768654 6709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "4a85e492c8154e9881a9cc2a105d7e64"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-fr9h"
I20240503 13:59:02.771776 6709 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 13:59:02.773838 6723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.774797 6709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:02.774911 6709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "4a85e492c8154e9881a9cc2a105d7e64"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-fr9h"
I20240503 13:59:02.774988 6709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:02.796116 6709 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:02.796753 6709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:02.796917 6709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:02.797461 6709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:02.798367 6709 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:02.798419 6709 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.798482 6709 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:02.798516 6709 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.807065 6709 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:46365
I20240503 13:59:02.807097 6836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:46365 every 8 connection(s)
I20240503 13:59:02.808043 6709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 13:59:02.814723 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 6709
I20240503 13:59:02.814814 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 13:59:02.816532 6837 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45485
I20240503 13:59:02.816751 6837 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:02.817011 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45485
--builtin_ntp_servers=127.0.99.148:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:02.817301 6837 heartbeater.cc:511] Master 127.0.99.190:45485 requested a full tablet report, sending...
I20240503 13:59:02.818315 6387 ts_manager.cc:194] Registered new tserver with Master: 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:02.818863 6387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:47343
W20240503 13:59:02.925391 6840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:02.925678 6840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:02.925771 6840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:02.929764 6840 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:02.929826 6840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:02.929894 6840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 13:59:02.933670 6840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45485
--never_fsync=true
--heap_profile_path=/tmp/kudu.6840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:02.934514 6840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:02.935611 6840 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:02.944468 6846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:02.945217 6848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:02.946071 6845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:02.946904 6840 server_base.cc:1034] running on GCE node
I20240503 13:59:02.947304 6840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:02.947865 6840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:02.949004 6840 hybrid_clock.cc:648] HybridClock initialized: now 1714744742948987 us; error 34 us; skew 500 ppm
I20240503 13:59:02.950834 6840 webserver.cc:460] Webserver started at http://127.0.99.132:35383/ using document root <none> and password file <none>
I20240503 13:59:02.951411 6840 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:02.951470 6840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:02.951633 6840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:02.953248 6840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "f58065db70b64ef8b1dc499aa6ae4460"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-fr9h"
I20240503 13:59:02.953712 6840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "f58065db70b64ef8b1dc499aa6ae4460"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-fr9h"
I20240503 13:59:02.956984 6840 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 13:59:02.959038 6854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.960018 6840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:02.960127 6840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "f58065db70b64ef8b1dc499aa6ae4460"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-fr9h"
I20240503 13:59:02.960204 6840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:02.981047 6840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:02.981619 6840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:02.981767 6840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:02.982259 6840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:02.983105 6840 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:02.983157 6840 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.983203 6840 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:02.983238 6840 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.991145 6840 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:43531
I20240503 13:59:02.991182 6967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:43531 every 8 connection(s)
I20240503 13:59:02.992136 6840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 13:59:02.992663 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 6840
I20240503 13:59:02.992785 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 13:59:02.994916 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45485
--builtin_ntp_servers=127.0.99.148:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:03.001492 6968 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45485
I20240503 13:59:03.001706 6968 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:03.002218 6968 heartbeater.cc:511] Master 127.0.99.190:45485 requested a full tablet report, sending...
I20240503 13:59:03.003331 6387 ts_manager.cc:194] Registered new tserver with Master: f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:03.004207 6387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:48051
W20240503 13:59:03.102634 6971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:03.102846 6971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:03.102890 6971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:03.105876 6971 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:03.105944 6971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:03.106010 6971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 13:59:03.109570 6971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45485
--never_fsync=true
--heap_profile_path=/tmp/kudu.6971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:03.110427 6971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:03.111616 6971 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:03.120455 6976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:03.121232 6977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:03.122025 6979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:03.122841 6971 server_base.cc:1034] running on GCE node
I20240503 13:59:03.123222 6971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:03.123845 6971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:03.125007 6971 hybrid_clock.cc:648] HybridClock initialized: now 1714744743124981 us; error 41 us; skew 500 ppm
I20240503 13:59:03.126978 6971 webserver.cc:460] Webserver started at http://127.0.99.133:37125/ using document root <none> and password file <none>
I20240503 13:59:03.127554 6971 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:03.127616 6971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:03.127779 6971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:03.129498 6971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "06ca8dbc004c42b99faa527a4e6f9545"
format_stamp: "Formatted at 2024-05-03 13:59:03 on dist-test-slave-fr9h"
I20240503 13:59:03.129940 6971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "06ca8dbc004c42b99faa527a4e6f9545"
format_stamp: "Formatted at 2024-05-03 13:59:03 on dist-test-slave-fr9h"
I20240503 13:59:03.133350 6971 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 13:59:03.135447 6985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:03.136471 6971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:03.136579 6971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "06ca8dbc004c42b99faa527a4e6f9545"
format_stamp: "Formatted at 2024-05-03 13:59:03 on dist-test-slave-fr9h"
I20240503 13:59:03.136674 6971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:03.160384 6971 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:03.161000 6971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:03.161283 6971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:03.161919 6971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:03.162935 6971 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:03.162986 6971 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:03.163045 6971 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:03.163077 6971 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:03.172637 6971 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:34547
I20240503 13:59:03.172672 7098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:34547 every 8 connection(s)
I20240503 13:59:03.173786 6971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 13:59:03.181396 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 6971
I20240503 13:59:03.181528 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 13:59:03.184087 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45485
--builtin_ntp_servers=127.0.99.148:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:03.186197 7099 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45485
I20240503 13:59:03.186436 7099 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:03.187081 7099 heartbeater.cc:511] Master 127.0.99.190:45485 requested a full tablet report, sending...
I20240503 13:59:03.188299 6387 ts_manager.cc:194] Registered new tserver with Master: 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:03.189114 6387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:59613
W20240503 13:59:03.311105 7102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:03.311370 7102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:03.311470 7102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:03.314512 7102 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:03.314589 7102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:03.314687 7102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 13:59:03.318497 7102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45485
--never_fsync=true
--heap_profile_path=/tmp/kudu.7102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:03.319476 7102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:03.320673 7102 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:03.329459 7107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:03.330210 7108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:03.331017 7110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:03.331766 7102 server_base.cc:1034] running on GCE node
I20240503 13:59:03.332151 7102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:03.332733 7102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:03.333880 7102 hybrid_clock.cc:648] HybridClock initialized: now 1714744743333860 us; error 34 us; skew 500 ppm
I20240503 13:59:03.335723 7102 webserver.cc:460] Webserver started at http://127.0.99.134:45163/ using document root <none> and password file <none>
I20240503 13:59:03.336252 7102 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:03.336305 7102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:03.336481 7102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:03.338119 7102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "8d574fb4a89d4eea947c859470d44dd4"
format_stamp: "Formatted at 2024-05-03 13:59:03 on dist-test-slave-fr9h"
I20240503 13:59:03.338587 7102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "8d574fb4a89d4eea947c859470d44dd4"
format_stamp: "Formatted at 2024-05-03 13:59:03 on dist-test-slave-fr9h"
I20240503 13:59:03.342222 7102 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240503 13:59:03.344408 7116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:03.345499 7102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:03.345613 7102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "8d574fb4a89d4eea947c859470d44dd4"
format_stamp: "Formatted at 2024-05-03 13:59:03 on dist-test-slave-fr9h"
I20240503 13:59:03.345711 7102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:03.362017 7102 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:03.362604 7102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:03.362756 7102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:03.363224 7102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:03.364116 7102 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:03.364166 7102 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:03.364225 7102 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:03.364250 7102 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:03.372808 7102 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:39481
I20240503 13:59:03.372835 7229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:39481 every 8 connection(s)
I20240503 13:59:03.373682 7102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 13:59:03.382015 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 7102
I20240503 13:59:03.382035 7230 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45485
I20240503 13:59:03.382164 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 13:59:03.382274 7230 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:03.382879 7230 heartbeater.cc:511] Master 127.0.99.190:45485 requested a full tablet report, sending...
I20240503 13:59:03.383813 6387 ts_manager.cc:194] Registered new tserver with Master: 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:03.384356 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45485
--builtin_ntp_servers=127.0.99.148:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:03.384516 6387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:58941
I20240503 13:59:03.463837 6575 heartbeater.cc:503] Master 127.0.99.190:45485 was elected leader, sending a full tablet report...
W20240503 13:59:03.491698 7233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:03.491910 7233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:03.491953 7233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:03.495009 7233 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:03.495082 7233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:03.495199 7233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 13:59:03.499179 7233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45485
--never_fsync=true
--heap_profile_path=/tmp/kudu.7233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:03.500152 7233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:03.501260 7233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:03.509975 7239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:03.510847 7238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:03.511714 7241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:03.512428 7233 server_base.cc:1034] running on GCE node
I20240503 13:59:03.512832 7233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:03.513411 7233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:03.514564 7233 hybrid_clock.cc:648] HybridClock initialized: now 1714744743514539 us; error 39 us; skew 500 ppm
I20240503 13:59:03.516680 7233 webserver.cc:460] Webserver started at http://127.0.99.135:37925/ using document root <none> and password file <none>
I20240503 13:59:03.517256 7233 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:03.517324 7233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:03.517503 7233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:03.519203 7233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "ab1d21935d6049888eb0ceccc3c9d735"
format_stamp: "Formatted at 2024-05-03 13:59:03 on dist-test-slave-fr9h"
I20240503 13:59:03.519727 7233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "ab1d21935d6049888eb0ceccc3c9d735"
format_stamp: "Formatted at 2024-05-03 13:59:03 on dist-test-slave-fr9h"
I20240503 13:59:03.523069 7233 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 13:59:03.525257 7247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:03.526383 7233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:03.526490 7233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "ab1d21935d6049888eb0ceccc3c9d735"
format_stamp: "Formatted at 2024-05-03 13:59:03 on dist-test-slave-fr9h"
I20240503 13:59:03.526588 7233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:03.543522 7233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:03.544111 7233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:03.544263 7233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:03.544763 7233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:03.545634 7233 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:03.545682 7233 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:03.545743 7233 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:03.545775 7233 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:03.554283 7233 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:34091
I20240503 13:59:03.554328 7360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:34091 every 8 connection(s)
I20240503 13:59:03.555213 7233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 13:59:03.559878 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 7233
I20240503 13:59:03.559993 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 13:59:03.562091 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45485
--builtin_ntp_servers=127.0.99.148:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:03.564108 7361 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45485
I20240503 13:59:03.564366 7361 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:03.564880 7361 heartbeater.cc:511] Master 127.0.99.190:45485 requested a full tablet report, sending...
I20240503 13:59:03.565862 6387 ts_manager.cc:194] Registered new tserver with Master: ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:03.566731 6387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:58597
I20240503 13:59:03.640537 6706 heartbeater.cc:503] Master 127.0.99.190:45485 was elected leader, sending a full tablet report...
W20240503 13:59:03.672334 7364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:03.672549 7364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:03.672591 7364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:03.675639 7364 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:03.675704 7364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:03.675776 7364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 13:59:03.679415 7364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45485
--never_fsync=true
--heap_profile_path=/tmp/kudu.7364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:03.680207 7364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:03.681238 7364 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:03.689920 7372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:03.690793 7370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:03.691524 7369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:03.692410 7364 server_base.cc:1034] running on GCE node
I20240503 13:59:03.692842 7364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:03.693441 7364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:03.694620 7364 hybrid_clock.cc:648] HybridClock initialized: now 1714744743694595 us; error 39 us; skew 500 ppm
I20240503 13:59:03.696738 7364 webserver.cc:460] Webserver started at http://127.0.99.136:36929/ using document root <none> and password file <none>
I20240503 13:59:03.697305 7364 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:03.697362 7364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:03.697531 7364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:03.699692 7364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "0d77752181e944698e809e325e5937e6"
format_stamp: "Formatted at 2024-05-03 13:59:03 on dist-test-slave-fr9h"
I20240503 13:59:03.700181 7364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "0d77752181e944698e809e325e5937e6"
format_stamp: "Formatted at 2024-05-03 13:59:03 on dist-test-slave-fr9h"
I20240503 13:59:03.703559 7364 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 13:59:03.705792 7378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:03.706889 7364 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:03.706992 7364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "0d77752181e944698e809e325e5937e6"
format_stamp: "Formatted at 2024-05-03 13:59:03 on dist-test-slave-fr9h"
I20240503 13:59:03.707095 7364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:03.725638 7364 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:03.726254 7364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:03.726402 7364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:03.726943 7364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:03.727977 7364 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:03.728025 7364 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:03.728068 7364 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:03.728101 7364 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:03.736778 7364 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:41931
I20240503 13:59:03.736804 7491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:41931 every 8 connection(s)
I20240503 13:59:03.737696 7364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 13:59:03.737988 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 7364
I20240503 13:59:03.738109 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 13:59:03.746706 7492 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45485
I20240503 13:59:03.746935 7492 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:03.747500 7492 heartbeater.cc:511] Master 127.0.99.190:45485 requested a full tablet report, sending...
I20240503 13:59:03.748459 6387 ts_manager.cc:194] Registered new tserver with Master: 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:03.749047 6387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:37123
I20240503 13:59:03.752267 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 13:59:03.761405 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 13:59:03.761494 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 6971
I20240503 13:59:03.767194 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 7102
I20240503 13:59:03.773217 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 7233
I20240503 13:59:03.778744 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu with pid 7364
I20240503 13:59:03.821348 6837 heartbeater.cc:503] Master 127.0.99.190:45485 was elected leader, sending a full tablet report...
I20240503 13:59:04.006069 6968 heartbeater.cc:503] Master 127.0.99.190:45485 was elected leader, sending a full tablet report...
I20240503 13:59:07.535841 6387 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:07.555724 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 7662baa6a6e5478f956632b6e5dd1290 (DEFAULT_TABLE table=rebalance_test_table_0 [id=41d8e6d525344c1dabeb32421a3ae38f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.556315 6508 tablet_service.cc:1467] Processing CreateTablet for tablet b50f9df84ee44a5a8b24d70a7c0e76d1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=41d8e6d525344c1dabeb32421a3ae38f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.556351 6771 tablet_service.cc:1467] Processing CreateTablet for tablet b50f9df84ee44a5a8b24d70a7c0e76d1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=41d8e6d525344c1dabeb32421a3ae38f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.556351 6901 tablet_service.cc:1467] Processing CreateTablet for tablet b0a4be5ebd3246b39b011743d9ae5daa (DEFAULT_TABLE table=rebalance_test_table_0 [id=41d8e6d525344c1dabeb32421a3ae38f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.556720 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7662baa6a6e5478f956632b6e5dd1290. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.557145 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b50f9df84ee44a5a8b24d70a7c0e76d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.557188 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b50f9df84ee44a5a8b24d70a7c0e76d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.557188 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0a4be5ebd3246b39b011743d9ae5daa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.559450 6638 tablet_service.cc:1467] Processing CreateTablet for tablet b50f9df84ee44a5a8b24d70a7c0e76d1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=41d8e6d525344c1dabeb32421a3ae38f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.559557 6509 tablet_service.cc:1467] Processing CreateTablet for tablet 7662baa6a6e5478f956632b6e5dd1290 (DEFAULT_TABLE table=rebalance_test_table_0 [id=41d8e6d525344c1dabeb32421a3ae38f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.559571 6902 tablet_service.cc:1467] Processing CreateTablet for tablet 7662baa6a6e5478f956632b6e5dd1290 (DEFAULT_TABLE table=rebalance_test_table_0 [id=41d8e6d525344c1dabeb32421a3ae38f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.559713 6638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b50f9df84ee44a5a8b24d70a7c0e76d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.559775 6509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7662baa6a6e5478f956632b6e5dd1290. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.559828 6902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7662baa6a6e5478f956632b6e5dd1290. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.562885 6770 tablet_service.cc:1467] Processing CreateTablet for tablet b0a4be5ebd3246b39b011743d9ae5daa (DEFAULT_TABLE table=rebalance_test_table_0 [id=41d8e6d525344c1dabeb32421a3ae38f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.563145 6770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0a4be5ebd3246b39b011743d9ae5daa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.564301 7512 tablet_bootstrap.cc:492] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:07.565341 6507 tablet_service.cc:1467] Processing CreateTablet for tablet b0a4be5ebd3246b39b011743d9ae5daa (DEFAULT_TABLE table=rebalance_test_table_0 [id=41d8e6d525344c1dabeb32421a3ae38f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.565615 6507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0a4be5ebd3246b39b011743d9ae5daa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.566455 7513 tablet_bootstrap.cc:492] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:07.566922 7512 tablet_bootstrap.cc:654] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.567780 7512 log.cc:824] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:07.568568 7513 tablet_bootstrap.cc:654] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.568984 7516 tablet_bootstrap.cc:492] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:07.569361 7513 log.cc:824] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 2145d15ab2d74c0287839266f1411a43: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:07.569572 7512 tablet_bootstrap.cc:492] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:07.569712 7514 tablet_bootstrap.cc:492] T 7662baa6a6e5478f956632b6e5dd1290 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:07.569738 7512 ts_tablet_manager.cc:1367] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 13:59:07.570917 7516 tablet_bootstrap.cc:654] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.571470 7514 tablet_bootstrap.cc:654] T 7662baa6a6e5478f956632b6e5dd1290 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.571693 7516 log.cc:824] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:07.572247 7514 log.cc:824] T 7662baa6a6e5478f956632b6e5dd1290 P 5f46e19644fd4f4b825c86ba6af768da: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:07.573258 7513 tablet_bootstrap.cc:492] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:07.573258 7516 tablet_bootstrap.cc:492] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:07.573419 7516 ts_tablet_manager.cc:1367] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 13:59:07.573427 7513 ts_tablet_manager.cc:1367] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 13:59:07.573843 7514 tablet_bootstrap.cc:492] T 7662baa6a6e5478f956632b6e5dd1290 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:07.573988 7514 ts_tablet_manager.cc:1367] T 7662baa6a6e5478f956632b6e5dd1290 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 13:59:07.574370 7512 raft_consensus.cc:348] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:07.574560 7512 raft_consensus.cc:374] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.574609 7512 raft_consensus.cc:729] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.575073 7512 consensus_queue.cc:260] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:07.575927 7512 ts_tablet_manager.cc:1398] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20240503 13:59:07.577152 7513 raft_consensus.cc:348] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:07.577152 7516 raft_consensus.cc:348] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:07.577172 7514 raft_consensus.cc:348] T 7662baa6a6e5478f956632b6e5dd1290 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:07.577333 7516 raft_consensus.cc:374] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.577333 7513 raft_consensus.cc:374] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.577368 7514 raft_consensus.cc:374] T 7662baa6a6e5478f956632b6e5dd1290 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.577381 7516 raft_consensus.cc:729] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.577381 7513 raft_consensus.cc:729] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.577425 7514 raft_consensus.cc:729] T 7662baa6a6e5478f956632b6e5dd1290 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.577805 7513 consensus_queue.cc:260] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:07.577805 7516 consensus_queue.cc:260] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:07.578377 7514 consensus_queue.cc:260] T 7662baa6a6e5478f956632b6e5dd1290 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:07.578656 7513 ts_tablet_manager.cc:1398] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20240503 13:59:07.578655 7516 ts_tablet_manager.cc:1398] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 13:59:07.579010 7513 tablet_bootstrap.cc:492] T 7662baa6a6e5478f956632b6e5dd1290 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:07.579372 7514 ts_tablet_manager.cc:1398] T 7662baa6a6e5478f956632b6e5dd1290 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20240503 13:59:07.580186 7513 tablet_bootstrap.cc:654] T 7662baa6a6e5478f956632b6e5dd1290 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.580240 7512 tablet_bootstrap.cc:492] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:07.580775 7514 tablet_bootstrap.cc:492] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:07.581149 7516 tablet_bootstrap.cc:492] T b0a4be5ebd3246b39b011743d9ae5daa P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:07.581758 7514 tablet_bootstrap.cc:654] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.582021 7512 tablet_bootstrap.cc:654] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.582134 7516 tablet_bootstrap.cc:654] T b0a4be5ebd3246b39b011743d9ae5daa P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.583086 7514 tablet_bootstrap.cc:492] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:07.583206 7514 ts_tablet_manager.cc:1367] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:07.583384 7516 tablet_bootstrap.cc:492] T b0a4be5ebd3246b39b011743d9ae5daa P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:07.583462 7516 ts_tablet_manager.cc:1367] T b0a4be5ebd3246b39b011743d9ae5daa P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:07.583648 7513 tablet_bootstrap.cc:492] T 7662baa6a6e5478f956632b6e5dd1290 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:07.583726 7513 ts_tablet_manager.cc:1367] T 7662baa6a6e5478f956632b6e5dd1290 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:07.583801 7514 raft_consensus.cc:348] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:07.583894 7514 raft_consensus.cc:374] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.583925 7514 raft_consensus.cc:729] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.583954 7516 raft_consensus.cc:348] T b0a4be5ebd3246b39b011743d9ae5daa P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:07.584064 7516 raft_consensus.cc:374] T b0a4be5ebd3246b39b011743d9ae5daa P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.584098 7516 raft_consensus.cc:729] T b0a4be5ebd3246b39b011743d9ae5daa P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.584082 7514 consensus_queue.cc:260] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:07.584244 7513 raft_consensus.cc:348] T 7662baa6a6e5478f956632b6e5dd1290 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:07.584338 7513 raft_consensus.cc:374] T 7662baa6a6e5478f956632b6e5dd1290 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.584371 7513 raft_consensus.cc:729] T 7662baa6a6e5478f956632b6e5dd1290 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.584367 7514 ts_tablet_manager.cc:1398] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:07.584445 7516 consensus_queue.cc:260] T b0a4be5ebd3246b39b011743d9ae5daa P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:07.584515 7514 tablet_bootstrap.cc:492] T b0a4be5ebd3246b39b011743d9ae5daa P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:07.584478 7513 consensus_queue.cc:260] T 7662baa6a6e5478f956632b6e5dd1290 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:07.584722 7516 ts_tablet_manager.cc:1398] T b0a4be5ebd3246b39b011743d9ae5daa P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:07.584753 7513 ts_tablet_manager.cc:1398] T 7662baa6a6e5478f956632b6e5dd1290 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:07.585333 7514 tablet_bootstrap.cc:654] T b0a4be5ebd3246b39b011743d9ae5daa P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.586308 7512 tablet_bootstrap.cc:492] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:07.586442 7512 ts_tablet_manager.cc:1367] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 13:59:07.587512 7512 raft_consensus.cc:348] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:07.587661 7512 raft_consensus.cc:374] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.587759 7512 raft_consensus.cc:729] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.587913 7512 consensus_queue.cc:260] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:07.588265 7512 ts_tablet_manager.cc:1398] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:07.588722 7514 tablet_bootstrap.cc:492] T b0a4be5ebd3246b39b011743d9ae5daa P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:07.588804 7514 ts_tablet_manager.cc:1367] T b0a4be5ebd3246b39b011743d9ae5daa P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:59:07.589253 7514 raft_consensus.cc:348] T b0a4be5ebd3246b39b011743d9ae5daa P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:07.589363 7514 raft_consensus.cc:374] T b0a4be5ebd3246b39b011743d9ae5daa P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.589416 7514 raft_consensus.cc:729] T b0a4be5ebd3246b39b011743d9ae5daa P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.589540 7514 consensus_queue.cc:260] T b0a4be5ebd3246b39b011743d9ae5daa P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:07.589803 7514 ts_tablet_manager.cc:1398] T b0a4be5ebd3246b39b011743d9ae5daa P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:07.630381 7520 raft_consensus.cc:482] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:07.630594 7520 raft_consensus.cc:504] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:07.630972 7521 raft_consensus.cc:482] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:07.631150 7521 raft_consensus.cc:504] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:07.631912 7520 leader_election.cc:288] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:07.632171 7521 leader_election.cc:288] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
W20240503 13:59:07.632809 6707 tablet.cc:2369] T 7662baa6a6e5478f956632b6e5dd1290 P 2145d15ab2d74c0287839266f1411a43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:59:07.637390 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0a4be5ebd3246b39b011743d9ae5daa" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:07.637586 6922 raft_consensus.cc:2454] T b0a4be5ebd3246b39b011743d9ae5daa P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:07.638060 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7662baa6a6e5478f956632b6e5dd1290" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:07.638128 6726 leader_election.cc:302] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:07.638329 6660 raft_consensus.cc:2454] T 7662baa6a6e5478f956632b6e5dd1290 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:07.638432 7520 raft_consensus.cc:2781] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:07.638402 6529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0a4be5ebd3246b39b011743d9ae5daa" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:07.638500 7520 raft_consensus.cc:482] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:07.638538 7520 raft_consensus.cc:3037] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:07.638648 6529 raft_consensus.cc:2454] T b0a4be5ebd3246b39b011743d9ae5daa P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:07.638711 6858 leader_election.cc:302] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:07.638909 7521 raft_consensus.cc:2781] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:07.638998 7521 raft_consensus.cc:482] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:07.639070 7521 raft_consensus.cc:3037] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:07.639987 7520 raft_consensus.cc:504] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:07.640444 7520 leader_election.cc:288] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:07.640460 6529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7662baa6a6e5478f956632b6e5dd1290" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:07.640583 6529 raft_consensus.cc:2454] T 7662baa6a6e5478f956632b6e5dd1290 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:07.640625 7521 raft_consensus.cc:504] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:07.640748 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0a4be5ebd3246b39b011743d9ae5daa" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:07.640821 6922 raft_consensus.cc:3037] T b0a4be5ebd3246b39b011743d9ae5daa P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:07.640913 6529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0a4be5ebd3246b39b011743d9ae5daa" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:07.641001 7521 leader_election.cc:288] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:07.641009 6529 raft_consensus.cc:3037] T b0a4be5ebd3246b39b011743d9ae5daa P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:07.641556 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7662baa6a6e5478f956632b6e5dd1290" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:07.641675 6660 raft_consensus.cc:3037] T 7662baa6a6e5478f956632b6e5dd1290 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:07.641701 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7662baa6a6e5478f956632b6e5dd1290" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:07.641763 6922 raft_consensus.cc:2454] T b0a4be5ebd3246b39b011743d9ae5daa P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:07.641795 6528 raft_consensus.cc:3037] T 7662baa6a6e5478f956632b6e5dd1290 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:07.642089 6529 raft_consensus.cc:2454] T b0a4be5ebd3246b39b011743d9ae5daa P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:07.642158 6726 leader_election.cc:302] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:07.642360 7520 raft_consensus.cc:2781] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:07.642596 7520 raft_consensus.cc:686] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:07.642724 6528 raft_consensus.cc:2454] T 7662baa6a6e5478f956632b6e5dd1290 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:07.642736 6660 raft_consensus.cc:2454] T 7662baa6a6e5478f956632b6e5dd1290 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:07.642958 7520 consensus_queue.cc:237] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:07.643028 6857 leader_election.cc:302] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:07.643190 7521 raft_consensus.cc:2781] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:07.643412 7521 raft_consensus.cc:686] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:07.643695 7521 consensus_queue.cc:237] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:07.644941 7520 raft_consensus.cc:482] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:07.645836 7520 raft_consensus.cc:504] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:07.646250 7520 leader_election.cc:288] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:07.646463 6387 catalog_manager.cc:5484] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:07.646288 6383 catalog_manager.cc:5484] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:07.647506 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50f9df84ee44a5a8b24d70a7c0e76d1" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:07.647629 6528 raft_consensus.cc:2454] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:07.647862 6726 leader_election.cc:302] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:07.648540 7520 raft_consensus.cc:2781] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:07.648615 7520 raft_consensus.cc:482] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:07.648671 7520 raft_consensus.cc:3037] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:07.649581 7520 raft_consensus.cc:504] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:07.649948 7520 leader_election.cc:288] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:07.650307 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50f9df84ee44a5a8b24d70a7c0e76d1" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:07.650421 6528 raft_consensus.cc:3037] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:07.650529 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50f9df84ee44a5a8b24d70a7c0e76d1" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:07.650547 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50f9df84ee44a5a8b24d70a7c0e76d1" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:07.650666 6660 raft_consensus.cc:2454] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:07.651232 6528 raft_consensus.cc:2454] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:07.651499 6726 leader_election.cc:302] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, 5f46e19644fd4f4b825c86ba6af768da; no voters: 2145d15ab2d74c0287839266f1411a43
I20240503 13:59:07.651661 7520 raft_consensus.cc:2781] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:07.651748 7520 raft_consensus.cc:686] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:07.651875 7520 consensus_queue.cc:237] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:07.653249 6387 catalog_manager.cc:5484] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:07.704186 6576 tablet.cc:2369] T 7662baa6a6e5478f956632b6e5dd1290 P 5f46e19644fd4f4b825c86ba6af768da: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 13:59:07.746402 6969 tablet.cc:2369] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 13:59:07.781569 7540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:07.781823 7540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:07.784974 7540 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
W20240503 13:59:07.817793 6838 tablet.cc:2369] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:59:07.879418 6528 raft_consensus.cc:1261] T 7662baa6a6e5478f956632b6e5dd1290 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:07.879772 6659 raft_consensus.cc:1261] T 7662baa6a6e5478f956632b6e5dd1290 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:07.880101 7521 consensus_queue.cc:1035] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:07.880517 7521 consensus_queue.cc:1035] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:07.880688 6659 raft_consensus.cc:3037] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:07.881834 6659 raft_consensus.cc:1261] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:07.882280 7520 consensus_queue.cc:1035] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:07.882344 6528 raft_consensus.cc:1261] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:07.882742 7520 consensus_queue.cc:1035] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:07.892732 6527 raft_consensus.cc:1261] T b0a4be5ebd3246b39b011743d9ae5daa P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:07.893240 6922 raft_consensus.cc:1261] T b0a4be5ebd3246b39b011743d9ae5daa P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:07.894198 7583 consensus_queue.cc:1035] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:07.894460 7534 consensus_queue.cc:1035] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:07.899017 7572 mvcc.cc:205] Tried to move back new op lower bound from 7023594487311982592 to 7023594486383235072. Current Snapshot: MvccSnapshot[applied={T|T < 7023594487311982592}]
I20240503 13:59:07.903391 7589 mvcc.cc:205] Tried to move back new op lower bound from 7023594487396782080 to 7023594486350958592. Current Snapshot: MvccSnapshot[applied={T|T < 7023594487396782080}]
I20240503 13:59:07.906286 7569 mvcc.cc:205] Tried to move back new op lower bound from 7023594487361478656 to 7023594486348566528. Current Snapshot: MvccSnapshot[applied={T|T < 7023594487361478656}]
I20240503 13:59:07.921028 7579 mvcc.cc:205] Tried to move back new op lower bound from 7023594487305027584 to 7023594486350958592. Current Snapshot: MvccSnapshot[applied={T|T < 7023594487305027584}]
INSERT report
rows total: 2000
time total: 124.747 ms
time per row: 0.0623733 ms
I20240503 13:59:07.956408 6378 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:07.961822 6640 tablet_service.cc:1467] Processing CreateTablet for tablet af98fddcb623403c841a77ecc10a3e07 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f757e59886674aafb5f975c30459df7e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.961846 6770 tablet_service.cc:1467] Processing CreateTablet for tablet af98fddcb623403c841a77ecc10a3e07 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f757e59886674aafb5f975c30459df7e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.962157 6770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af98fddcb623403c841a77ecc10a3e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.962157 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af98fddcb623403c841a77ecc10a3e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.964236 7513 tablet_bootstrap.cc:492] T af98fddcb623403c841a77ecc10a3e07 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:07.964280 7512 tablet_bootstrap.cc:492] T af98fddcb623403c841a77ecc10a3e07 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:07.964561 6901 tablet_service.cc:1467] Processing CreateTablet for tablet af98fddcb623403c841a77ecc10a3e07 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f757e59886674aafb5f975c30459df7e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.964826 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af98fddcb623403c841a77ecc10a3e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.964846 6638 tablet_service.cc:1467] Processing CreateTablet for tablet 9de4c2ac66344f719b3fe56ffbcc804d (DEFAULT_TABLE table=rebalance_test_table_1 [id=f757e59886674aafb5f975c30459df7e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.965063 6638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9de4c2ac66344f719b3fe56ffbcc804d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.965165 7512 tablet_bootstrap.cc:654] T af98fddcb623403c841a77ecc10a3e07 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.965164 7513 tablet_bootstrap.cc:654] T af98fddcb623403c841a77ecc10a3e07 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.966205 7512 tablet_bootstrap.cc:492] T af98fddcb623403c841a77ecc10a3e07 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:07.966302 7512 ts_tablet_manager.cc:1367] T af98fddcb623403c841a77ecc10a3e07 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:07.966773 7516 tablet_bootstrap.cc:492] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:07.966737 7512 raft_consensus.cc:348] T af98fddcb623403c841a77ecc10a3e07 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:07.966836 7512 raft_consensus.cc:374] T af98fddcb623403c841a77ecc10a3e07 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.966868 7512 raft_consensus.cc:729] T af98fddcb623403c841a77ecc10a3e07 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.966989 7512 consensus_queue.cc:260] T af98fddcb623403c841a77ecc10a3e07 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:07.967152 6902 tablet_service.cc:1467] Processing CreateTablet for tablet 350ba659df804b95b0692091dc2f68a2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f757e59886674aafb5f975c30459df7e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.967255 7512 ts_tablet_manager.cc:1398] T af98fddcb623403c841a77ecc10a3e07 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:07.967579 6507 tablet_service.cc:1467] Processing CreateTablet for tablet 9de4c2ac66344f719b3fe56ffbcc804d (DEFAULT_TABLE table=rebalance_test_table_1 [id=f757e59886674aafb5f975c30459df7e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.967666 7516 tablet_bootstrap.cc:654] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.967846 6507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9de4c2ac66344f719b3fe56ffbcc804d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.967994 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 350ba659df804b95b0692091dc2f68a2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f757e59886674aafb5f975c30459df7e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.968202 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 350ba659df804b95b0692091dc2f68a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.970041 7514 tablet_bootstrap.cc:492] T 9de4c2ac66344f719b3fe56ffbcc804d P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:07.970541 6902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 350ba659df804b95b0692091dc2f68a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.970676 6771 tablet_service.cc:1467] Processing CreateTablet for tablet 9de4c2ac66344f719b3fe56ffbcc804d (DEFAULT_TABLE table=rebalance_test_table_1 [id=f757e59886674aafb5f975c30459df7e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.970890 7514 tablet_bootstrap.cc:654] T 9de4c2ac66344f719b3fe56ffbcc804d P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.970927 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9de4c2ac66344f719b3fe56ffbcc804d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.971845 7516 tablet_bootstrap.cc:492] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:07.971923 7516 ts_tablet_manager.cc:1367] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:07.972383 7516 raft_consensus.cc:348] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:07.972412 6770 tablet_service.cc:1467] Processing CreateTablet for tablet 350ba659df804b95b0692091dc2f68a2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f757e59886674aafb5f975c30459df7e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.972478 7516 raft_consensus.cc:374] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.972515 7516 raft_consensus.cc:729] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.972622 6770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 350ba659df804b95b0692091dc2f68a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.972668 7516 consensus_queue.cc:260] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:07.972841 7512 tablet_bootstrap.cc:492] T 9de4c2ac66344f719b3fe56ffbcc804d P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:07.973026 7516 ts_tablet_manager.cc:1398] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:07.973203 7516 tablet_bootstrap.cc:492] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:07.973533 7514 tablet_bootstrap.cc:492] T 9de4c2ac66344f719b3fe56ffbcc804d P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:07.973611 7514 ts_tablet_manager.cc:1367] T 9de4c2ac66344f719b3fe56ffbcc804d P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:59:07.973667 7512 tablet_bootstrap.cc:654] T 9de4c2ac66344f719b3fe56ffbcc804d P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.974015 7516 tablet_bootstrap.cc:654] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.974021 7514 raft_consensus.cc:348] T 9de4c2ac66344f719b3fe56ffbcc804d P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:07.974117 7514 raft_consensus.cc:374] T 9de4c2ac66344f719b3fe56ffbcc804d P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.974153 7514 raft_consensus.cc:729] T 9de4c2ac66344f719b3fe56ffbcc804d P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.974295 7514 consensus_queue.cc:260] T 9de4c2ac66344f719b3fe56ffbcc804d P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:07.974567 7514 ts_tablet_manager.cc:1398] T 9de4c2ac66344f719b3fe56ffbcc804d P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:07.975026 7516 tablet_bootstrap.cc:492] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:07.975101 7516 ts_tablet_manager.cc:1367] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:07.975543 7516 raft_consensus.cc:348] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:07.975651 7516 raft_consensus.cc:374] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.975698 7516 raft_consensus.cc:729] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.975826 7516 consensus_queue.cc:260] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:07.976111 7516 ts_tablet_manager.cc:1398] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:07.976359 7513 tablet_bootstrap.cc:492] T af98fddcb623403c841a77ecc10a3e07 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:07.976424 7513 ts_tablet_manager.cc:1367] T af98fddcb623403c841a77ecc10a3e07 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20240503 13:59:07.976842 7513 raft_consensus.cc:348] T af98fddcb623403c841a77ecc10a3e07 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:07.976940 7513 raft_consensus.cc:374] T af98fddcb623403c841a77ecc10a3e07 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.976961 7513 raft_consensus.cc:729] T af98fddcb623403c841a77ecc10a3e07 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.977066 7513 consensus_queue.cc:260] T af98fddcb623403c841a77ecc10a3e07 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:07.977174 7512 tablet_bootstrap.cc:492] T 9de4c2ac66344f719b3fe56ffbcc804d P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:07.977245 7512 ts_tablet_manager.cc:1367] T 9de4c2ac66344f719b3fe56ffbcc804d P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:07.977322 7513 ts_tablet_manager.cc:1398] T af98fddcb623403c841a77ecc10a3e07 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:07.977478 7513 tablet_bootstrap.cc:492] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:07.977656 7512 raft_consensus.cc:348] T 9de4c2ac66344f719b3fe56ffbcc804d P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:07.977756 7512 raft_consensus.cc:374] T 9de4c2ac66344f719b3fe56ffbcc804d P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.977811 7512 raft_consensus.cc:729] T 9de4c2ac66344f719b3fe56ffbcc804d P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.977936 7512 consensus_queue.cc:260] T 9de4c2ac66344f719b3fe56ffbcc804d P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:07.978210 7512 ts_tablet_manager.cc:1398] T 9de4c2ac66344f719b3fe56ffbcc804d P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:07.978325 7513 tablet_bootstrap.cc:654] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.978363 7512 tablet_bootstrap.cc:492] T 350ba659df804b95b0692091dc2f68a2 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:07.979239 7513 tablet_bootstrap.cc:492] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:07.979241 7512 tablet_bootstrap.cc:654] T 350ba659df804b95b0692091dc2f68a2 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.979352 7513 ts_tablet_manager.cc:1367] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 13:59:07.979797 7513 raft_consensus.cc:348] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:07.979894 7513 raft_consensus.cc:374] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.979928 7513 raft_consensus.cc:729] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.980064 7513 consensus_queue.cc:260] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:07.980393 7513 ts_tablet_manager.cc:1398] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:07.980558 7513 tablet_bootstrap.cc:492] T 350ba659df804b95b0692091dc2f68a2 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:07.981132 7512 tablet_bootstrap.cc:492] T 350ba659df804b95b0692091dc2f68a2 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:07.981226 7512 ts_tablet_manager.cc:1367] T 350ba659df804b95b0692091dc2f68a2 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:07.981463 7513 tablet_bootstrap.cc:654] T 350ba659df804b95b0692091dc2f68a2 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.981679 7512 raft_consensus.cc:348] T 350ba659df804b95b0692091dc2f68a2 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:07.981781 7512 raft_consensus.cc:374] T 350ba659df804b95b0692091dc2f68a2 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.981833 7512 raft_consensus.cc:729] T 350ba659df804b95b0692091dc2f68a2 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.981982 7512 consensus_queue.cc:260] T 350ba659df804b95b0692091dc2f68a2 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:07.982321 7512 ts_tablet_manager.cc:1398] T 350ba659df804b95b0692091dc2f68a2 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:07.983019 7513 tablet_bootstrap.cc:492] T 350ba659df804b95b0692091dc2f68a2 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:07.983114 7513 ts_tablet_manager.cc:1367] T 350ba659df804b95b0692091dc2f68a2 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:07.983471 7513 raft_consensus.cc:348] T 350ba659df804b95b0692091dc2f68a2 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:07.983557 7513 raft_consensus.cc:374] T 350ba659df804b95b0692091dc2f68a2 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.983587 7513 raft_consensus.cc:729] T 350ba659df804b95b0692091dc2f68a2 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.983695 7513 consensus_queue.cc:260] T 350ba659df804b95b0692091dc2f68a2 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:07.983930 7513 ts_tablet_manager.cc:1398] T 350ba659df804b95b0692091dc2f68a2 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:08.002898 7521 raft_consensus.cc:482] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:08.003010 7521 raft_consensus.cc:504] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:08.003612 7521 leader_election.cc:288] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:08.003945 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af98fddcb623403c841a77ecc10a3e07" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:08.004077 6660 raft_consensus.cc:2454] T af98fddcb623403c841a77ecc10a3e07 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:08.004395 6858 leader_election.cc:302] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:08.004555 7521 raft_consensus.cc:2781] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:08.004598 7521 raft_consensus.cc:482] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:08.004631 7521 raft_consensus.cc:3037] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:08.005573 7521 raft_consensus.cc:504] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:08.005897 7521 leader_election.cc:288] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:08.006189 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af98fddcb623403c841a77ecc10a3e07" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:08.006300 6660 raft_consensus.cc:3037] T af98fddcb623403c841a77ecc10a3e07 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:08.007355 6660 raft_consensus.cc:2454] T af98fddcb623403c841a77ecc10a3e07 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:08.007603 6858 leader_election.cc:302] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:08.007768 7521 raft_consensus.cc:2781] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:08.007834 7521 raft_consensus.cc:686] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:08.007781 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af98fddcb623403c841a77ecc10a3e07" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:08.007933 7521 consensus_queue.cc:237] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:08.007979 6791 raft_consensus.cc:2454] T af98fddcb623403c841a77ecc10a3e07 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:08.008217 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af98fddcb623403c841a77ecc10a3e07" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:08.008297 6790 raft_consensus.cc:3037] T af98fddcb623403c841a77ecc10a3e07 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:08.009239 6790 raft_consensus.cc:2454] T af98fddcb623403c841a77ecc10a3e07 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:08.010213 6383 catalog_manager.cc:5484] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:08.086150 7535 raft_consensus.cc:482] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:08.086295 7535 raft_consensus.cc:504] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:08.086737 7535 leader_election.cc:288] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:08.087146 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350ba659df804b95b0692091dc2f68a2" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:08.087168 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350ba659df804b95b0692091dc2f68a2" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:08.087312 6660 raft_consensus.cc:2454] T 350ba659df804b95b0692091dc2f68a2 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:08.087324 6790 raft_consensus.cc:2454] T 350ba659df804b95b0692091dc2f68a2 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:08.087628 6858 leader_election.cc:302] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:08.087798 7535 raft_consensus.cc:2781] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:08.087852 7535 raft_consensus.cc:482] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:08.087877 7535 raft_consensus.cc:3037] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:08.088797 7535 raft_consensus.cc:504] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:08.089157 7535 leader_election.cc:288] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:08.089455 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350ba659df804b95b0692091dc2f68a2" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:08.089478 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350ba659df804b95b0692091dc2f68a2" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:08.089565 6660 raft_consensus.cc:3037] T 350ba659df804b95b0692091dc2f68a2 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:08.089574 6790 raft_consensus.cc:3037] T 350ba659df804b95b0692091dc2f68a2 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:08.090291 6790 raft_consensus.cc:2454] T 350ba659df804b95b0692091dc2f68a2 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:08.090291 6660 raft_consensus.cc:2454] T 350ba659df804b95b0692091dc2f68a2 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:08.090549 6856 leader_election.cc:302] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:08.090755 7535 raft_consensus.cc:2781] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:08.090837 7535 raft_consensus.cc:686] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:08.090981 7535 consensus_queue.cc:237] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:08.092548 6383 catalog_manager.cc:5484] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:08.125141 7574 raft_consensus.cc:482] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:08.125273 7574 raft_consensus.cc:504] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:08.126291 7574 leader_election.cc:288] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:08.129925 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9de4c2ac66344f719b3fe56ffbcc804d" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:08.129926 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9de4c2ac66344f719b3fe56ffbcc804d" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:08.130064 6528 raft_consensus.cc:2454] T 9de4c2ac66344f719b3fe56ffbcc804d P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:08.130075 6790 raft_consensus.cc:2454] T 9de4c2ac66344f719b3fe56ffbcc804d P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:08.130419 6594 leader_election.cc:302] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:08.130647 7574 raft_consensus.cc:2781] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:08.130710 7574 raft_consensus.cc:482] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:08.130748 7574 raft_consensus.cc:3037] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:08.131644 7574 raft_consensus.cc:504] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:08.131942 7574 leader_election.cc:288] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:08.132241 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9de4c2ac66344f719b3fe56ffbcc804d" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:08.132256 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9de4c2ac66344f719b3fe56ffbcc804d" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:08.132342 6790 raft_consensus.cc:3037] T 9de4c2ac66344f719b3fe56ffbcc804d P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:08.132352 6528 raft_consensus.cc:3037] T 9de4c2ac66344f719b3fe56ffbcc804d P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:08.133031 6790 raft_consensus.cc:2454] T 9de4c2ac66344f719b3fe56ffbcc804d P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:08.133090 6528 raft_consensus.cc:2454] T 9de4c2ac66344f719b3fe56ffbcc804d P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:08.133298 6594 leader_election.cc:302] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:08.133435 7574 raft_consensus.cc:2781] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:08.133525 7574 raft_consensus.cc:686] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 13:59:08.133827 7574 consensus_queue.cc:237] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:08.136247 6383 catalog_manager.cc:5484] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:08.268002 7616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:08.268254 7616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:08.271252 7616 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 13:59:08.348821 6660 raft_consensus.cc:1261] T 350ba659df804b95b0692091dc2f68a2 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:08.348984 6790 raft_consensus.cc:1261] T 350ba659df804b95b0692091dc2f68a2 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:08.349573 7521 consensus_queue.cc:1035] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:08.350108 7535 consensus_queue.cc:1035] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:08.354614 6659 raft_consensus.cc:1261] T af98fddcb623403c841a77ecc10a3e07 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:08.354626 6789 raft_consensus.cc:1261] T af98fddcb623403c841a77ecc10a3e07 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:08.355167 7521 consensus_queue.cc:1035] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:08.355454 7521 consensus_queue.cc:1035] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:08.355604 6791 raft_consensus.cc:1261] T 9de4c2ac66344f719b3fe56ffbcc804d P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:08.357367 6528 raft_consensus.cc:1261] T 9de4c2ac66344f719b3fe56ffbcc804d P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:08.357865 7574 consensus_queue.cc:1035] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:08.358395 7574 consensus_queue.cc:1035] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 98.6826 ms
time per row: 0.0493413 ms
I20240503 13:59:08.413640 6381 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:08.418917 6902 tablet_service.cc:1467] Processing CreateTablet for tablet 12d82a64f2f543f582d154ac47c0f0cb (DEFAULT_TABLE table=rebalance_test_table_2 [id=fb49f602527d4645be66ec1c72a6702d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.419000 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 12d82a64f2f543f582d154ac47c0f0cb (DEFAULT_TABLE table=rebalance_test_table_2 [id=fb49f602527d4645be66ec1c72a6702d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.419234 6902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12d82a64f2f543f582d154ac47c0f0cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.419258 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12d82a64f2f543f582d154ac47c0f0cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.420090 6507 tablet_service.cc:1467] Processing CreateTablet for tablet 12d82a64f2f543f582d154ac47c0f0cb (DEFAULT_TABLE table=rebalance_test_table_2 [id=fb49f602527d4645be66ec1c72a6702d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.420378 6507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12d82a64f2f543f582d154ac47c0f0cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.420776 6509 tablet_service.cc:1467] Processing CreateTablet for tablet 0e889726631646499b8c11742ee0185f (DEFAULT_TABLE table=rebalance_test_table_2 [id=fb49f602527d4645be66ec1c72a6702d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.420960 6509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e889726631646499b8c11742ee0185f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.421319 7516 tablet_bootstrap.cc:492] T 12d82a64f2f543f582d154ac47c0f0cb P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:08.422024 7516 tablet_bootstrap.cc:654] T 12d82a64f2f543f582d154ac47c0f0cb P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.422346 7514 tablet_bootstrap.cc:492] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:08.422502 7513 tablet_bootstrap.cc:492] T 12d82a64f2f543f582d154ac47c0f0cb P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:08.422744 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 94eeec82221e48f6918a2f56e7aa5f37 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fb49f602527d4645be66ec1c72a6702d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.422952 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94eeec82221e48f6918a2f56e7aa5f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.423264 7514 tablet_bootstrap.cc:654] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.423748 6770 tablet_service.cc:1467] Processing CreateTablet for tablet 94eeec82221e48f6918a2f56e7aa5f37 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fb49f602527d4645be66ec1c72a6702d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.424013 6770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94eeec82221e48f6918a2f56e7aa5f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.424047 6901 tablet_service.cc:1467] Processing CreateTablet for tablet 0e889726631646499b8c11742ee0185f (DEFAULT_TABLE table=rebalance_test_table_2 [id=fb49f602527d4645be66ec1c72a6702d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.424244 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e889726631646499b8c11742ee0185f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.424932 6902 tablet_service.cc:1467] Processing CreateTablet for tablet 94eeec82221e48f6918a2f56e7aa5f37 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fb49f602527d4645be66ec1c72a6702d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.425181 6902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94eeec82221e48f6918a2f56e7aa5f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.426416 7512 tablet_bootstrap.cc:492] T 94eeec82221e48f6918a2f56e7aa5f37 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:08.423269 7513 tablet_bootstrap.cc:654] T 12d82a64f2f543f582d154ac47c0f0cb P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.427258 7512 tablet_bootstrap.cc:654] T 94eeec82221e48f6918a2f56e7aa5f37 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.427554 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 0e889726631646499b8c11742ee0185f (DEFAULT_TABLE table=rebalance_test_table_2 [id=fb49f602527d4645be66ec1c72a6702d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.427614 7516 tablet_bootstrap.cc:492] T 12d82a64f2f543f582d154ac47c0f0cb P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:08.427685 7516 ts_tablet_manager.cc:1367] T 12d82a64f2f543f582d154ac47c0f0cb P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 13:59:08.427779 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e889726631646499b8c11742ee0185f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.427826 7513 tablet_bootstrap.cc:492] T 12d82a64f2f543f582d154ac47c0f0cb P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:08.427910 7513 ts_tablet_manager.cc:1367] T 12d82a64f2f543f582d154ac47c0f0cb P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 13:59:08.428141 7516 raft_consensus.cc:348] T 12d82a64f2f543f582d154ac47c0f0cb P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:08.428234 7516 raft_consensus.cc:374] T 12d82a64f2f543f582d154ac47c0f0cb P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.428262 7516 raft_consensus.cc:729] T 12d82a64f2f543f582d154ac47c0f0cb P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.428390 7512 tablet_bootstrap.cc:492] T 94eeec82221e48f6918a2f56e7aa5f37 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:08.428344 7513 raft_consensus.cc:348] T 12d82a64f2f543f582d154ac47c0f0cb P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:08.428375 7516 consensus_queue.cc:260] T 12d82a64f2f543f582d154ac47c0f0cb P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:08.428440 7513 raft_consensus.cc:374] T 12d82a64f2f543f582d154ac47c0f0cb P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.428458 7512 ts_tablet_manager.cc:1367] T 94eeec82221e48f6918a2f56e7aa5f37 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:08.428476 7513 raft_consensus.cc:729] T 12d82a64f2f543f582d154ac47c0f0cb P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.428596 7513 consensus_queue.cc:260] T 12d82a64f2f543f582d154ac47c0f0cb P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:08.428632 7516 ts_tablet_manager.cc:1398] T 12d82a64f2f543f582d154ac47c0f0cb P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:08.428815 7516 tablet_bootstrap.cc:492] T 0e889726631646499b8c11742ee0185f P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:08.428857 7513 ts_tablet_manager.cc:1398] T 12d82a64f2f543f582d154ac47c0f0cb P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:08.428855 7512 raft_consensus.cc:348] T 94eeec82221e48f6918a2f56e7aa5f37 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:08.428951 7512 raft_consensus.cc:374] T 94eeec82221e48f6918a2f56e7aa5f37 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.428989 7512 raft_consensus.cc:729] T 94eeec82221e48f6918a2f56e7aa5f37 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.429105 7512 consensus_queue.cc:260] T 94eeec82221e48f6918a2f56e7aa5f37 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:08.429383 7512 ts_tablet_manager.cc:1398] T 94eeec82221e48f6918a2f56e7aa5f37 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:08.429677 7516 tablet_bootstrap.cc:654] T 0e889726631646499b8c11742ee0185f P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.430719 7516 tablet_bootstrap.cc:492] T 0e889726631646499b8c11742ee0185f P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:08.430812 7516 ts_tablet_manager.cc:1367] T 0e889726631646499b8c11742ee0185f P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:08.431210 7516 raft_consensus.cc:348] T 0e889726631646499b8c11742ee0185f P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:08.431340 7516 raft_consensus.cc:374] T 0e889726631646499b8c11742ee0185f P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.431375 7516 raft_consensus.cc:729] T 0e889726631646499b8c11742ee0185f P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.432169 7514 tablet_bootstrap.cc:492] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:08.432261 7514 ts_tablet_manager.cc:1367] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240503 13:59:08.432691 7514 raft_consensus.cc:348] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:08.432828 7514 raft_consensus.cc:374] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.432873 7514 raft_consensus.cc:729] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.433022 7514 consensus_queue.cc:260] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:08.433162 7516 consensus_queue.cc:260] T 0e889726631646499b8c11742ee0185f P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:08.433250 7514 ts_tablet_manager.cc:1398] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:08.433419 7514 tablet_bootstrap.cc:492] T 0e889726631646499b8c11742ee0185f P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:08.433753 7516 ts_tablet_manager.cc:1398] T 0e889726631646499b8c11742ee0185f P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:08.433897 7516 tablet_bootstrap.cc:492] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:08.434312 7514 tablet_bootstrap.cc:654] T 0e889726631646499b8c11742ee0185f P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.434916 7513 tablet_bootstrap.cc:492] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:08.435405 7514 tablet_bootstrap.cc:492] T 0e889726631646499b8c11742ee0185f P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:08.435506 7514 ts_tablet_manager.cc:1367] T 0e889726631646499b8c11742ee0185f P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:08.435781 7513 tablet_bootstrap.cc:654] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.436326 7514 raft_consensus.cc:348] T 0e889726631646499b8c11742ee0185f P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:08.436445 7514 raft_consensus.cc:374] T 0e889726631646499b8c11742ee0185f P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.436491 7514 raft_consensus.cc:729] T 0e889726631646499b8c11742ee0185f P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.436622 7514 consensus_queue.cc:260] T 0e889726631646499b8c11742ee0185f P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:08.436977 7516 tablet_bootstrap.cc:654] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.436981 7514 ts_tablet_manager.cc:1398] T 0e889726631646499b8c11742ee0185f P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:08.437136 7514 tablet_bootstrap.cc:492] T 94eeec82221e48f6918a2f56e7aa5f37 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:08.437234 7513 tablet_bootstrap.cc:492] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:08.437314 7513 ts_tablet_manager.cc:1367] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:08.437726 7513 raft_consensus.cc:348] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:08.437811 7513 raft_consensus.cc:374] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.437846 7513 raft_consensus.cc:729] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.437968 7513 consensus_queue.cc:260] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:08.438022 7514 tablet_bootstrap.cc:654] T 94eeec82221e48f6918a2f56e7aa5f37 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.438230 7513 ts_tablet_manager.cc:1398] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:08.438556 7516 tablet_bootstrap.cc:492] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:08.438632 7516 ts_tablet_manager.cc:1367] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:59:08.439087 7514 tablet_bootstrap.cc:492] T 94eeec82221e48f6918a2f56e7aa5f37 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:08.439038 7516 raft_consensus.cc:348] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:08.439145 7516 raft_consensus.cc:374] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.439163 7514 ts_tablet_manager.cc:1367] T 94eeec82221e48f6918a2f56e7aa5f37 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:08.439178 7516 raft_consensus.cc:729] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.439332 7516 consensus_queue.cc:260] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:08.439648 7516 ts_tablet_manager.cc:1398] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:08.439648 7514 raft_consensus.cc:348] T 94eeec82221e48f6918a2f56e7aa5f37 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:08.439742 7514 raft_consensus.cc:374] T 94eeec82221e48f6918a2f56e7aa5f37 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.439783 7514 raft_consensus.cc:729] T 94eeec82221e48f6918a2f56e7aa5f37 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.439890 7514 consensus_queue.cc:260] T 94eeec82221e48f6918a2f56e7aa5f37 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:08.440157 7514 ts_tablet_manager.cc:1398] T 94eeec82221e48f6918a2f56e7aa5f37 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:08.483398 7571 raft_consensus.cc:482] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:08.483542 7571 raft_consensus.cc:504] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:08.484596 7571 leader_election.cc:288] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:08.488718 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12d82a64f2f543f582d154ac47c0f0cb" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:08.488862 6922 raft_consensus.cc:2454] T 12d82a64f2f543f582d154ac47c0f0cb P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:08.488806 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12d82a64f2f543f582d154ac47c0f0cb" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:08.488974 6660 raft_consensus.cc:2454] T 12d82a64f2f543f582d154ac47c0f0cb P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:08.489212 6464 leader_election.cc:302] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:08.489480 7571 raft_consensus.cc:2781] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:08.489537 7571 raft_consensus.cc:482] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:08.489564 7571 raft_consensus.cc:3037] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:08.490415 7571 raft_consensus.cc:504] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:08.490631 7574 raft_consensus.cc:482] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:08.490715 7574 raft_consensus.cc:504] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:08.490770 7571 leader_election.cc:288] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:08.491067 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12d82a64f2f543f582d154ac47c0f0cb" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:08.491086 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12d82a64f2f543f582d154ac47c0f0cb" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:08.491183 6922 raft_consensus.cc:3037] T 12d82a64f2f543f582d154ac47c0f0cb P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:08.491142 7574 leader_election.cc:288] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:08.491183 6660 raft_consensus.cc:3037] T 12d82a64f2f543f582d154ac47c0f0cb P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:08.491467 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e889726631646499b8c11742ee0185f" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:08.491575 6528 raft_consensus.cc:2454] T 0e889726631646499b8c11742ee0185f P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:08.491878 6595 leader_election.cc:302] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:08.492038 7574 raft_consensus.cc:2781] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:08.492082 7574 raft_consensus.cc:482] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:08.492110 7574 raft_consensus.cc:3037] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:08.492185 6922 raft_consensus.cc:2454] T 12d82a64f2f543f582d154ac47c0f0cb P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:08.492293 6660 raft_consensus.cc:2454] T 12d82a64f2f543f582d154ac47c0f0cb P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:08.492530 6464 leader_election.cc:302] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:08.492921 7571 raft_consensus.cc:2781] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:08.493086 7571 raft_consensus.cc:686] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Becoming Leader. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Running, Role: LEADER
I20240503 13:59:08.493057 7574 raft_consensus.cc:504] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:08.493381 7574 leader_election.cc:288] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:08.493407 7571 consensus_queue.cc:237] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:08.493675 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e889726631646499b8c11742ee0185f" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:08.493754 6528 raft_consensus.cc:3037] T 0e889726631646499b8c11742ee0185f P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:08.494573 6528 raft_consensus.cc:2454] T 0e889726631646499b8c11742ee0185f P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:08.494845 6595 leader_election.cc:302] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:08.495033 7574 raft_consensus.cc:2781] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:08.495141 7574 raft_consensus.cc:686] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 13:59:08.495260 7574 consensus_queue.cc:237] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:08.496351 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e889726631646499b8c11742ee0185f" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:08.496412 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e889726631646499b8c11742ee0185f" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:08.496472 6922 raft_consensus.cc:2454] T 0e889726631646499b8c11742ee0185f P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:08.496819 6379 catalog_manager.cc:5484] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:08.497290 6383 catalog_manager.cc:5484] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:08.577059 7521 raft_consensus.cc:482] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:08.577206 7521 raft_consensus.cc:504] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:08.577620 7521 leader_election.cc:288] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:08.578004 6789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94eeec82221e48f6918a2f56e7aa5f37" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:08.578004 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94eeec82221e48f6918a2f56e7aa5f37" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:08.578140 6528 raft_consensus.cc:2454] T 94eeec82221e48f6918a2f56e7aa5f37 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:08.578155 6789 raft_consensus.cc:2454] T 94eeec82221e48f6918a2f56e7aa5f37 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:08.578464 6857 leader_election.cc:302] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:08.578626 7521 raft_consensus.cc:2781] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:08.578686 7521 raft_consensus.cc:482] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:08.578714 7521 raft_consensus.cc:3037] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:08.579584 7521 raft_consensus.cc:504] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:08.579905 7521 leader_election.cc:288] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:08.580185 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94eeec82221e48f6918a2f56e7aa5f37" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:08.580219 6789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94eeec82221e48f6918a2f56e7aa5f37" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:08.580312 6789 raft_consensus.cc:3037] T 94eeec82221e48f6918a2f56e7aa5f37 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:08.580317 6528 raft_consensus.cc:3037] T 94eeec82221e48f6918a2f56e7aa5f37 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:08.581069 6789 raft_consensus.cc:2454] T 94eeec82221e48f6918a2f56e7aa5f37 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:08.581070 6528 raft_consensus.cc:2454] T 94eeec82221e48f6918a2f56e7aa5f37 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:08.581341 6856 leader_election.cc:302] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:08.581532 7521 raft_consensus.cc:2781] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:08.581602 7521 raft_consensus.cc:686] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:08.581774 7521 consensus_queue.cc:237] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:08.583527 6383 catalog_manager.cc:5484] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:08.717255 7668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:08.717504 7668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:08.720523 7668 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 13:59:08.814424 6789 raft_consensus.cc:1261] T 94eeec82221e48f6918a2f56e7aa5f37 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:08.814891 7535 consensus_queue.cc:1035] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:08.816623 6528 raft_consensus.cc:1261] T 94eeec82221e48f6918a2f56e7aa5f37 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:08.816989 7535 consensus_queue.cc:1035] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:08.821739 6921 raft_consensus.cc:3037] T 0e889726631646499b8c11742ee0185f P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:08.822072 6529 raft_consensus.cc:1261] T 0e889726631646499b8c11742ee0185f P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:08.822467 7522 consensus_queue.cc:1035] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:08.823364 6921 raft_consensus.cc:1261] T 0e889726631646499b8c11742ee0185f P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:08.824115 7522 consensus_queue.cc:1035] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:08.838137 6922 raft_consensus.cc:1261] T 12d82a64f2f543f582d154ac47c0f0cb P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:08.838423 6660 raft_consensus.cc:1261] T 12d82a64f2f543f582d154ac47c0f0cb P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:08.839385 7571 consensus_queue.cc:1035] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:08.839814 7571 consensus_queue.cc:1035] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 127.75 ms
time per row: 0.0638748 ms
I20240503 13:59:08.890408 6378 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:08.895931 6640 tablet_service.cc:1467] Processing CreateTablet for tablet e106b1432baf423493c95fcce63a55da (DEFAULT_TABLE table=rebalance_test_table_3 [id=0ea567ca26f349a788891ba7d2d3751a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.895988 6770 tablet_service.cc:1467] Processing CreateTablet for tablet e106b1432baf423493c95fcce63a55da (DEFAULT_TABLE table=rebalance_test_table_3 [id=0ea567ca26f349a788891ba7d2d3751a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.896250 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e106b1432baf423493c95fcce63a55da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.896222 6771 tablet_service.cc:1467] Processing CreateTablet for tablet 3331108688a3485a9c44c65600f0f3e5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0ea567ca26f349a788891ba7d2d3751a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.896250 6770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e106b1432baf423493c95fcce63a55da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.896490 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3331108688a3485a9c44c65600f0f3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.897980 7512 tablet_bootstrap.cc:492] T 3331108688a3485a9c44c65600f0f3e5 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:08.898677 6640 tablet_service.cc:1467] Processing CreateTablet for tablet d6af833fca454d0d826c376c95cf1a76 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0ea567ca26f349a788891ba7d2d3751a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.898890 7512 tablet_bootstrap.cc:654] T 3331108688a3485a9c44c65600f0f3e5 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.898953 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6af833fca454d0d826c376c95cf1a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.899196 6770 tablet_service.cc:1467] Processing CreateTablet for tablet d6af833fca454d0d826c376c95cf1a76 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0ea567ca26f349a788891ba7d2d3751a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.899425 6770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6af833fca454d0d826c376c95cf1a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.899396 6509 tablet_service.cc:1467] Processing CreateTablet for tablet 3331108688a3485a9c44c65600f0f3e5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0ea567ca26f349a788891ba7d2d3751a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.899505 7513 tablet_bootstrap.cc:492] T e106b1432baf423493c95fcce63a55da P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:08.899653 6509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3331108688a3485a9c44c65600f0f3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.900089 6508 tablet_service.cc:1467] Processing CreateTablet for tablet e106b1432baf423493c95fcce63a55da (DEFAULT_TABLE table=rebalance_test_table_3 [id=0ea567ca26f349a788891ba7d2d3751a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.900303 7513 tablet_bootstrap.cc:654] T e106b1432baf423493c95fcce63a55da P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.900329 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e106b1432baf423493c95fcce63a55da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.901340 7513 tablet_bootstrap.cc:492] T e106b1432baf423493c95fcce63a55da P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:08.901417 7513 ts_tablet_manager.cc:1367] T e106b1432baf423493c95fcce63a55da P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:08.901566 7514 tablet_bootstrap.cc:492] T 3331108688a3485a9c44c65600f0f3e5 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:08.901787 7513 raft_consensus.cc:348] T e106b1432baf423493c95fcce63a55da P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:08.901903 7513 raft_consensus.cc:374] T e106b1432baf423493c95fcce63a55da P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.901938 7513 raft_consensus.cc:729] T e106b1432baf423493c95fcce63a55da P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.902052 7513 consensus_queue.cc:260] T e106b1432baf423493c95fcce63a55da P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:08.902339 7513 ts_tablet_manager.cc:1398] T e106b1432baf423493c95fcce63a55da P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 13:59:08.902392 7514 tablet_bootstrap.cc:654] T 3331108688a3485a9c44c65600f0f3e5 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.902336 6902 tablet_service.cc:1467] Processing CreateTablet for tablet 3331108688a3485a9c44c65600f0f3e5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0ea567ca26f349a788891ba7d2d3751a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.902498 7513 tablet_bootstrap.cc:492] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:08.902630 6902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3331108688a3485a9c44c65600f0f3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.903509 7513 tablet_bootstrap.cc:654] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.904450 7513 tablet_bootstrap.cc:492] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:08.904541 7513 ts_tablet_manager.cc:1367] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:08.904939 7516 tablet_bootstrap.cc:492] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:08.905119 6901 tablet_service.cc:1467] Processing CreateTablet for tablet d6af833fca454d0d826c376c95cf1a76 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0ea567ca26f349a788891ba7d2d3751a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.905354 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6af833fca454d0d826c376c95cf1a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.905563 7512 tablet_bootstrap.cc:492] T 3331108688a3485a9c44c65600f0f3e5 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:08.905652 7512 ts_tablet_manager.cc:1367] T 3331108688a3485a9c44c65600f0f3e5 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 13:59:08.905747 7513 raft_consensus.cc:348] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:08.905884 7513 raft_consensus.cc:374] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.905917 7513 raft_consensus.cc:729] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.906016 7513 consensus_queue.cc:260] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:08.906059 7512 raft_consensus.cc:348] T 3331108688a3485a9c44c65600f0f3e5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:08.906162 7516 tablet_bootstrap.cc:654] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.906167 7512 raft_consensus.cc:374] T 3331108688a3485a9c44c65600f0f3e5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.906200 7512 raft_consensus.cc:729] T 3331108688a3485a9c44c65600f0f3e5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.906301 7513 ts_tablet_manager.cc:1398] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:08.906335 7512 consensus_queue.cc:260] T 3331108688a3485a9c44c65600f0f3e5 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:08.906599 7512 ts_tablet_manager.cc:1398] T 3331108688a3485a9c44c65600f0f3e5 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:08.907436 7516 tablet_bootstrap.cc:492] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:08.907517 7516 ts_tablet_manager.cc:1367] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:08.907946 7516 raft_consensus.cc:348] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:08.908041 7516 raft_consensus.cc:374] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.908077 7516 raft_consensus.cc:729] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.908186 7516 consensus_queue.cc:260] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:08.908385 7514 tablet_bootstrap.cc:492] T 3331108688a3485a9c44c65600f0f3e5 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:08.908473 7514 ts_tablet_manager.cc:1367] T 3331108688a3485a9c44c65600f0f3e5 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 13:59:08.908705 7516 ts_tablet_manager.cc:1398] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:08.908864 7516 tablet_bootstrap.cc:492] T d6af833fca454d0d826c376c95cf1a76 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:08.908897 7514 raft_consensus.cc:348] T 3331108688a3485a9c44c65600f0f3e5 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:08.909008 7514 raft_consensus.cc:374] T 3331108688a3485a9c44c65600f0f3e5 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.909061 7514 raft_consensus.cc:729] T 3331108688a3485a9c44c65600f0f3e5 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.909198 7514 consensus_queue.cc:260] T 3331108688a3485a9c44c65600f0f3e5 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:08.909462 7514 ts_tablet_manager.cc:1398] T 3331108688a3485a9c44c65600f0f3e5 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:08.909502 7512 tablet_bootstrap.cc:492] T e106b1432baf423493c95fcce63a55da P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:08.909638 7514 tablet_bootstrap.cc:492] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:08.909718 7516 tablet_bootstrap.cc:654] T d6af833fca454d0d826c376c95cf1a76 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.910442 7514 tablet_bootstrap.cc:654] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.911046 7516 tablet_bootstrap.cc:492] T d6af833fca454d0d826c376c95cf1a76 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:08.911125 7516 ts_tablet_manager.cc:1367] T d6af833fca454d0d826c376c95cf1a76 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:08.911545 7516 raft_consensus.cc:348] T d6af833fca454d0d826c376c95cf1a76 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:08.911660 7516 raft_consensus.cc:374] T d6af833fca454d0d826c376c95cf1a76 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.911693 7516 raft_consensus.cc:729] T d6af833fca454d0d826c376c95cf1a76 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.911821 7516 consensus_queue.cc:260] T d6af833fca454d0d826c376c95cf1a76 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:08.912091 7516 ts_tablet_manager.cc:1398] T d6af833fca454d0d826c376c95cf1a76 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:08.913053 7512 tablet_bootstrap.cc:654] T e106b1432baf423493c95fcce63a55da P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.913302 7514 tablet_bootstrap.cc:492] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:08.913390 7514 ts_tablet_manager.cc:1367] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 13:59:08.913872 7514 raft_consensus.cc:348] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:08.913969 7514 raft_consensus.cc:374] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.914002 7514 raft_consensus.cc:729] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.914068 7512 tablet_bootstrap.cc:492] T e106b1432baf423493c95fcce63a55da P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:08.914134 7512 ts_tablet_manager.cc:1367] T e106b1432baf423493c95fcce63a55da P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:08.914110 7514 consensus_queue.cc:260] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:08.914362 7514 ts_tablet_manager.cc:1398] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:08.914566 7512 raft_consensus.cc:348] T e106b1432baf423493c95fcce63a55da P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:08.914667 7512 raft_consensus.cc:374] T e106b1432baf423493c95fcce63a55da P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.914705 7512 raft_consensus.cc:729] T e106b1432baf423493c95fcce63a55da P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.914817 7512 consensus_queue.cc:260] T e106b1432baf423493c95fcce63a55da P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:08.915184 7512 ts_tablet_manager.cc:1398] T e106b1432baf423493c95fcce63a55da P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:08.915343 7512 tablet_bootstrap.cc:492] T d6af833fca454d0d826c376c95cf1a76 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:08.916043 7512 tablet_bootstrap.cc:654] T d6af833fca454d0d826c376c95cf1a76 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.917915 7512 tablet_bootstrap.cc:492] T d6af833fca454d0d826c376c95cf1a76 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:08.917994 7512 ts_tablet_manager.cc:1367] T d6af833fca454d0d826c376c95cf1a76 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:08.918393 7512 raft_consensus.cc:348] T d6af833fca454d0d826c376c95cf1a76 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:08.918475 7512 raft_consensus.cc:374] T d6af833fca454d0d826c376c95cf1a76 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.918509 7512 raft_consensus.cc:729] T d6af833fca454d0d826c376c95cf1a76 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.918627 7512 consensus_queue.cc:260] T d6af833fca454d0d826c376c95cf1a76 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:08.918918 7512 ts_tablet_manager.cc:1398] T d6af833fca454d0d826c376c95cf1a76 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:09.003005 7706 raft_consensus.cc:482] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:09.003131 7706 raft_consensus.cc:504] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:09.003605 7706 leader_election.cc:288] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:09.003978 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6af833fca454d0d826c376c95cf1a76" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:09.004027 6789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6af833fca454d0d826c376c95cf1a76" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:09.004107 6921 raft_consensus.cc:2454] T d6af833fca454d0d826c376c95cf1a76 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:09.004177 6789 raft_consensus.cc:2454] T d6af833fca454d0d826c376c95cf1a76 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:09.004400 6595 leader_election.cc:302] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:09.004565 7706 raft_consensus.cc:2781] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:09.004608 7706 raft_consensus.cc:482] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:09.004639 7706 raft_consensus.cc:3037] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.005609 7706 raft_consensus.cc:504] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:09.005944 7706 leader_election.cc:288] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:09.006230 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6af833fca454d0d826c376c95cf1a76" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:09.006345 6921 raft_consensus.cc:3037] T d6af833fca454d0d826c376c95cf1a76 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.006465 6789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6af833fca454d0d826c376c95cf1a76" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:09.006559 6789 raft_consensus.cc:3037] T d6af833fca454d0d826c376c95cf1a76 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.007493 6921 raft_consensus.cc:2454] T d6af833fca454d0d826c376c95cf1a76 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:09.007588 6789 raft_consensus.cc:2454] T d6af833fca454d0d826c376c95cf1a76 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:09.007865 6595 leader_election.cc:302] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:09.008041 7706 raft_consensus.cc:2781] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:09.008132 7706 raft_consensus.cc:686] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 13:59:09.008258 7706 consensus_queue.cc:237] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:09.009893 6385 catalog_manager.cc:5484] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:09.023603 7663 raft_consensus.cc:482] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:09.023757 7663 raft_consensus.cc:504] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:09.024292 7663 leader_election.cc:288] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:09.024770 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e106b1432baf423493c95fcce63a55da" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:09.024931 6660 raft_consensus.cc:2454] T e106b1432baf423493c95fcce63a55da P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:09.025270 6465 leader_election.cc:302] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:09.025449 7663 raft_consensus.cc:2781] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:09.025502 7663 raft_consensus.cc:482] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:09.025533 7663 raft_consensus.cc:3037] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.026407 7663 raft_consensus.cc:504] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:09.026744 7663 leader_election.cc:288] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Requested vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:09.027045 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e106b1432baf423493c95fcce63a55da" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:09.027091 7701 raft_consensus.cc:482] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:09.027170 6660 raft_consensus.cc:3037] T e106b1432baf423493c95fcce63a55da P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.027184 7701 raft_consensus.cc:504] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:09.027621 7701 leader_election.cc:288] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:09.028080 6789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3331108688a3485a9c44c65600f0f3e5" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:09.028183 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3331108688a3485a9c44c65600f0f3e5" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:09.028234 6789 raft_consensus.cc:2454] T 3331108688a3485a9c44c65600f0f3e5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:09.028234 6660 raft_consensus.cc:2454] T e106b1432baf423493c95fcce63a55da P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:09.028323 6528 raft_consensus.cc:2454] T 3331108688a3485a9c44c65600f0f3e5 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:09.028445 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e106b1432baf423493c95fcce63a55da" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:09.028587 6790 raft_consensus.cc:3037] T e106b1432baf423493c95fcce63a55da P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.028445 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e106b1432baf423493c95fcce63a55da" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:09.028620 6856 leader_election.cc:302] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:09.028659 6465 leader_election.cc:302] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:09.028983 7663 raft_consensus.cc:2781] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:09.029073 7663 raft_consensus.cc:686] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Becoming Leader. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Running, Role: LEADER
I20240503 13:59:09.029209 7663 consensus_queue.cc:237] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:09.029578 6790 raft_consensus.cc:2454] T e106b1432baf423493c95fcce63a55da P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:09.029911 7701 raft_consensus.cc:2781] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:09.029987 7701 raft_consensus.cc:482] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:09.030033 7701 raft_consensus.cc:3037] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.030938 7701 raft_consensus.cc:504] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:09.031258 7701 leader_election.cc:288] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:09.031172 6378 catalog_manager.cc:5484] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:09.031579 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3331108688a3485a9c44c65600f0f3e5" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:09.031569 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3331108688a3485a9c44c65600f0f3e5" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:09.031682 6790 raft_consensus.cc:3037] T 3331108688a3485a9c44c65600f0f3e5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.031682 6528 raft_consensus.cc:3037] T 3331108688a3485a9c44c65600f0f3e5 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.032521 6528 raft_consensus.cc:2454] T 3331108688a3485a9c44c65600f0f3e5 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:09.032513 6790 raft_consensus.cc:2454] T 3331108688a3485a9c44c65600f0f3e5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:09.033361 6857 leader_election.cc:302] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:09.033562 7701 raft_consensus.cc:2781] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:09.033661 7701 raft_consensus.cc:686] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:09.033789 7701 consensus_queue.cc:237] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:09.035352 6378 catalog_manager.cc:5484] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } }
W20240503 13:59:09.165611 7724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:09.165825 7724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:09.168794 7724 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 13:59:09.250973 6921 raft_consensus.cc:1261] T d6af833fca454d0d826c376c95cf1a76 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:09.251442 7706 consensus_queue.cc:1035] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:09.253667 6790 raft_consensus.cc:1261] T e106b1432baf423493c95fcce63a55da P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:09.254051 6660 raft_consensus.cc:1261] T e106b1432baf423493c95fcce63a55da P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:09.254698 7663 consensus_queue.cc:1035] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:09.258092 7663 consensus_queue.cc:1035] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:09.258800 6790 raft_consensus.cc:1261] T d6af833fca454d0d826c376c95cf1a76 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:09.259158 7706 consensus_queue.cc:1035] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:09.259903 6789 raft_consensus.cc:1261] T 3331108688a3485a9c44c65600f0f3e5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:09.260299 7521 consensus_queue.cc:1035] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:09.261562 6528 raft_consensus.cc:1261] T 3331108688a3485a9c44c65600f0f3e5 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:09.265111 7701 consensus_queue.cc:1035] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 118.121 ms
time per row: 0.0590606 ms
I20240503 13:59:09.326627 6380 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:09.331992 6770 tablet_service.cc:1467] Processing CreateTablet for tablet 712bb27624184da8ba9ba0d1248abc02 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1b29a48257a84332b723b8a0a9ce625b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:09.331976 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 712bb27624184da8ba9ba0d1248abc02 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1b29a48257a84332b723b8a0a9ce625b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:09.332283 6770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 712bb27624184da8ba9ba0d1248abc02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:09.332295 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 712bb27624184da8ba9ba0d1248abc02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:09.332866 6902 tablet_service.cc:1467] Processing CreateTablet for tablet ae76780e32bb4e5cb4e1c86403544173 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1b29a48257a84332b723b8a0a9ce625b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:09.332866 6901 tablet_service.cc:1467] Processing CreateTablet for tablet 63ec50dbf2624c24bc0e9325ab28bb86 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1b29a48257a84332b723b8a0a9ce625b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:09.333204 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63ec50dbf2624c24bc0e9325ab28bb86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:09.333408 6902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae76780e32bb4e5cb4e1c86403544173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:09.334390 7512 tablet_bootstrap.cc:492] T 712bb27624184da8ba9ba0d1248abc02 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:09.334578 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 712bb27624184da8ba9ba0d1248abc02 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1b29a48257a84332b723b8a0a9ce625b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:09.334828 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 712bb27624184da8ba9ba0d1248abc02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:09.335243 7512 tablet_bootstrap.cc:654] T 712bb27624184da8ba9ba0d1248abc02 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:09.335261 7516 tablet_bootstrap.cc:492] T ae76780e32bb4e5cb4e1c86403544173 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:09.336154 7516 tablet_bootstrap.cc:654] T ae76780e32bb4e5cb4e1c86403544173 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:09.336621 6770 tablet_service.cc:1467] Processing CreateTablet for tablet ae76780e32bb4e5cb4e1c86403544173 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1b29a48257a84332b723b8a0a9ce625b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:09.336863 6770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae76780e32bb4e5cb4e1c86403544173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:09.337002 7514 tablet_bootstrap.cc:492] T 712bb27624184da8ba9ba0d1248abc02 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:09.337088 7513 tablet_bootstrap.cc:492] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:09.337188 7516 tablet_bootstrap.cc:492] T ae76780e32bb4e5cb4e1c86403544173 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:09.337255 7516 ts_tablet_manager.cc:1367] T ae76780e32bb4e5cb4e1c86403544173 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:09.337657 7516 raft_consensus.cc:348] T ae76780e32bb4e5cb4e1c86403544173 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:09.337781 7514 tablet_bootstrap.cc:654] T 712bb27624184da8ba9ba0d1248abc02 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:09.337777 7516 raft_consensus.cc:374] T ae76780e32bb4e5cb4e1c86403544173 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:09.337826 7516 raft_consensus.cc:729] T ae76780e32bb4e5cb4e1c86403544173 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:09.337910 7513 tablet_bootstrap.cc:654] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:09.337951 7516 consensus_queue.cc:260] T ae76780e32bb4e5cb4e1c86403544173 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:09.338224 7516 ts_tablet_manager.cc:1398] T ae76780e32bb4e5cb4e1c86403544173 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:09.338392 7516 tablet_bootstrap.cc:492] T 63ec50dbf2624c24bc0e9325ab28bb86 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:09.338501 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 63ec50dbf2624c24bc0e9325ab28bb86 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1b29a48257a84332b723b8a0a9ce625b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:09.338716 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63ec50dbf2624c24bc0e9325ab28bb86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:09.338860 6509 tablet_service.cc:1467] Processing CreateTablet for tablet ae76780e32bb4e5cb4e1c86403544173 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1b29a48257a84332b723b8a0a9ce625b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:09.339068 6509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae76780e32bb4e5cb4e1c86403544173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:09.339187 7516 tablet_bootstrap.cc:654] T 63ec50dbf2624c24bc0e9325ab28bb86 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:09.340951 7513 tablet_bootstrap.cc:492] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:09.341040 7513 ts_tablet_manager.cc:1367] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:59:09.341476 7513 raft_consensus.cc:348] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:09.341572 7513 raft_consensus.cc:374] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:09.341609 7513 raft_consensus.cc:729] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:09.341734 7513 consensus_queue.cc:260] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:09.341754 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 63ec50dbf2624c24bc0e9325ab28bb86 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1b29a48257a84332b723b8a0a9ce625b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:09.341935 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63ec50dbf2624c24bc0e9325ab28bb86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:09.342018 7513 ts_tablet_manager.cc:1398] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:09.342136 7513 tablet_bootstrap.cc:492] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:09.342883 7513 tablet_bootstrap.cc:654] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:09.344273 7516 tablet_bootstrap.cc:492] T 63ec50dbf2624c24bc0e9325ab28bb86 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:09.344367 7516 ts_tablet_manager.cc:1367] T 63ec50dbf2624c24bc0e9325ab28bb86 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 13:59:09.344455 7514 tablet_bootstrap.cc:492] T 712bb27624184da8ba9ba0d1248abc02 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:09.344565 7514 ts_tablet_manager.cc:1367] T 712bb27624184da8ba9ba0d1248abc02 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240503 13:59:09.344789 7516 raft_consensus.cc:348] T 63ec50dbf2624c24bc0e9325ab28bb86 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:09.344897 7516 raft_consensus.cc:374] T 63ec50dbf2624c24bc0e9325ab28bb86 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:09.344929 7516 raft_consensus.cc:729] T 63ec50dbf2624c24bc0e9325ab28bb86 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:09.344962 7514 raft_consensus.cc:348] T 712bb27624184da8ba9ba0d1248abc02 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:09.345079 7514 raft_consensus.cc:374] T 712bb27624184da8ba9ba0d1248abc02 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:09.345052 7516 consensus_queue.cc:260] T 63ec50dbf2624c24bc0e9325ab28bb86 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:09.345113 7514 raft_consensus.cc:729] T 712bb27624184da8ba9ba0d1248abc02 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:09.345324 7516 ts_tablet_manager.cc:1398] T 63ec50dbf2624c24bc0e9325ab28bb86 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:09.345230 7514 consensus_queue.cc:260] T 712bb27624184da8ba9ba0d1248abc02 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:09.345676 7514 ts_tablet_manager.cc:1398] T 712bb27624184da8ba9ba0d1248abc02 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:09.345858 7514 tablet_bootstrap.cc:492] T ae76780e32bb4e5cb4e1c86403544173 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:09.346287 7512 tablet_bootstrap.cc:492] T 712bb27624184da8ba9ba0d1248abc02 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:09.346383 7512 ts_tablet_manager.cc:1367] T 712bb27624184da8ba9ba0d1248abc02 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 13:59:09.346660 7514 tablet_bootstrap.cc:654] T ae76780e32bb4e5cb4e1c86403544173 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:09.346832 7512 raft_consensus.cc:348] T 712bb27624184da8ba9ba0d1248abc02 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:09.346928 7512 raft_consensus.cc:374] T 712bb27624184da8ba9ba0d1248abc02 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:09.346964 7512 raft_consensus.cc:729] T 712bb27624184da8ba9ba0d1248abc02 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:09.347085 7512 consensus_queue.cc:260] T 712bb27624184da8ba9ba0d1248abc02 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:09.347445 7512 ts_tablet_manager.cc:1398] T 712bb27624184da8ba9ba0d1248abc02 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:09.347635 7512 tablet_bootstrap.cc:492] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:09.348086 7513 tablet_bootstrap.cc:492] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:09.348170 7513 ts_tablet_manager.cc:1367] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 13:59:09.348500 7512 tablet_bootstrap.cc:654] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:09.348709 7513 raft_consensus.cc:348] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:09.348853 7513 raft_consensus.cc:374] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:09.348894 7513 raft_consensus.cc:729] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:09.349050 7513 consensus_queue.cc:260] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:09.349313 7513 ts_tablet_manager.cc:1398] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:09.350502 7514 tablet_bootstrap.cc:492] T ae76780e32bb4e5cb4e1c86403544173 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:09.350584 7514 ts_tablet_manager.cc:1367] T ae76780e32bb4e5cb4e1c86403544173 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240503 13:59:09.351107 7514 raft_consensus.cc:348] T ae76780e32bb4e5cb4e1c86403544173 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:09.351195 7514 raft_consensus.cc:374] T ae76780e32bb4e5cb4e1c86403544173 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:09.351229 7514 raft_consensus.cc:729] T ae76780e32bb4e5cb4e1c86403544173 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:09.351385 7514 consensus_queue.cc:260] T ae76780e32bb4e5cb4e1c86403544173 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:09.351704 7514 ts_tablet_manager.cc:1398] T ae76780e32bb4e5cb4e1c86403544173 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:09.351862 7514 tablet_bootstrap.cc:492] T 63ec50dbf2624c24bc0e9325ab28bb86 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:09.352648 7514 tablet_bootstrap.cc:654] T 63ec50dbf2624c24bc0e9325ab28bb86 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:09.353608 7514 tablet_bootstrap.cc:492] T 63ec50dbf2624c24bc0e9325ab28bb86 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:09.353667 7514 ts_tablet_manager.cc:1367] T 63ec50dbf2624c24bc0e9325ab28bb86 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:09.353672 7512 tablet_bootstrap.cc:492] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:09.353747 7512 ts_tablet_manager.cc:1367] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 13:59:09.353987 7514 raft_consensus.cc:348] T 63ec50dbf2624c24bc0e9325ab28bb86 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:09.354068 7514 raft_consensus.cc:374] T 63ec50dbf2624c24bc0e9325ab28bb86 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:09.354097 7514 raft_consensus.cc:729] T 63ec50dbf2624c24bc0e9325ab28bb86 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:09.354188 7512 raft_consensus.cc:348] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:09.354236 7514 consensus_queue.cc:260] T 63ec50dbf2624c24bc0e9325ab28bb86 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:09.354300 7512 raft_consensus.cc:374] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:09.354353 7512 raft_consensus.cc:729] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:09.354485 7514 ts_tablet_manager.cc:1398] T 63ec50dbf2624c24bc0e9325ab28bb86 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:09.354463 7512 consensus_queue.cc:260] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:09.354703 7512 ts_tablet_manager.cc:1398] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:09.428105 7758 raft_consensus.cc:482] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:09.428256 7758 raft_consensus.cc:504] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:09.428738 7758 leader_election.cc:288] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:09.429107 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "712bb27624184da8ba9ba0d1248abc02" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:09.429163 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "712bb27624184da8ba9ba0d1248abc02" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:09.429273 6528 raft_consensus.cc:2454] T 712bb27624184da8ba9ba0d1248abc02 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:09.429301 6791 raft_consensus.cc:2454] T 712bb27624184da8ba9ba0d1248abc02 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:09.429601 6595 leader_election.cc:302] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:09.429790 7758 raft_consensus.cc:2781] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:09.429848 7758 raft_consensus.cc:482] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:09.429875 7758 raft_consensus.cc:3037] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.430743 7758 raft_consensus.cc:504] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:09.431039 7758 leader_election.cc:288] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:09.431370 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "712bb27624184da8ba9ba0d1248abc02" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:09.431388 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "712bb27624184da8ba9ba0d1248abc02" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:09.431492 6791 raft_consensus.cc:3037] T 712bb27624184da8ba9ba0d1248abc02 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.431491 6528 raft_consensus.cc:3037] T 712bb27624184da8ba9ba0d1248abc02 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.432459 6791 raft_consensus.cc:2454] T 712bb27624184da8ba9ba0d1248abc02 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:09.432451 6528 raft_consensus.cc:2454] T 712bb27624184da8ba9ba0d1248abc02 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:09.432744 6594 leader_election.cc:302] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:09.432896 7758 raft_consensus.cc:2781] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:09.433007 7758 raft_consensus.cc:686] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 13:59:09.433140 7758 consensus_queue.cc:237] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:09.434743 6381 catalog_manager.cc:5484] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:09.577153 7758 raft_consensus.cc:482] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:09.577284 7758 raft_consensus.cc:504] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:09.577714 7758 leader_election.cc:288] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:09.578125 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ec50dbf2624c24bc0e9325ab28bb86" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:09.578197 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ec50dbf2624c24bc0e9325ab28bb86" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:09.578281 6921 raft_consensus.cc:2454] T 63ec50dbf2624c24bc0e9325ab28bb86 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:09.578315 6528 raft_consensus.cc:2454] T 63ec50dbf2624c24bc0e9325ab28bb86 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:09.578642 6595 leader_election.cc:302] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:09.578819 7758 raft_consensus.cc:2781] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:09.578881 7758 raft_consensus.cc:482] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:09.578919 7758 raft_consensus.cc:3037] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.579820 7758 raft_consensus.cc:504] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:09.580139 7758 leader_election.cc:288] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:09.580118 7520 raft_consensus.cc:482] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:09.580225 7520 raft_consensus.cc:504] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:09.580516 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ec50dbf2624c24bc0e9325ab28bb86" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:09.580516 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ec50dbf2624c24bc0e9325ab28bb86" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:09.580614 7520 leader_election.cc:288] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:09.580629 6528 raft_consensus.cc:3037] T 63ec50dbf2624c24bc0e9325ab28bb86 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.580648 6921 raft_consensus.cc:3037] T 63ec50dbf2624c24bc0e9325ab28bb86 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.580909 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae76780e32bb4e5cb4e1c86403544173" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:09.581027 6922 raft_consensus.cc:2454] T ae76780e32bb4e5cb4e1c86403544173 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:09.581307 6726 leader_election.cc:302] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:09.581468 7520 raft_consensus.cc:2781] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:09.581521 7520 raft_consensus.cc:482] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:09.581569 7520 raft_consensus.cc:3037] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.581560 6528 raft_consensus.cc:2454] T 63ec50dbf2624c24bc0e9325ab28bb86 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:09.581609 6921 raft_consensus.cc:2454] T 63ec50dbf2624c24bc0e9325ab28bb86 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:09.581836 6595 leader_election.cc:302] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:09.581975 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae76780e32bb4e5cb4e1c86403544173" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:09.582005 7758 raft_consensus.cc:2781] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:09.582072 6528 raft_consensus.cc:2454] T ae76780e32bb4e5cb4e1c86403544173 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:09.582100 7758 raft_consensus.cc:686] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 13:59:09.582372 7758 consensus_queue.cc:237] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:09.582510 7520 raft_consensus.cc:504] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:09.582825 7520 leader_election.cc:288] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:09.583175 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae76780e32bb4e5cb4e1c86403544173" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:09.583170 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae76780e32bb4e5cb4e1c86403544173" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:09.583319 6921 raft_consensus.cc:3037] T ae76780e32bb4e5cb4e1c86403544173 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.583416 6528 raft_consensus.cc:3037] T ae76780e32bb4e5cb4e1c86403544173 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.584184 6921 raft_consensus.cc:2454] T ae76780e32bb4e5cb4e1c86403544173 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:09.584184 6528 raft_consensus.cc:2454] T ae76780e32bb4e5cb4e1c86403544173 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:09.584465 6726 leader_election.cc:302] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:09.584393 6381 catalog_manager.cc:5484] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:09.584748 7520 raft_consensus.cc:2781] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:09.584877 7520 raft_consensus.cc:686] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:09.585191 7520 consensus_queue.cc:237] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:09.586720 6386 catalog_manager.cc:5484] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:09.729570 7777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:09.729846 7777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:09.733198 7777 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 13:59:09.818542 6528 raft_consensus.cc:1261] T 63ec50dbf2624c24bc0e9325ab28bb86 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:09.818945 6921 raft_consensus.cc:1261] T 63ec50dbf2624c24bc0e9325ab28bb86 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:09.819375 7758 consensus_queue.cc:1035] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:09.819505 7574 consensus_queue.cc:1035] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:09.820255 6922 raft_consensus.cc:1261] T ae76780e32bb4e5cb4e1c86403544173 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:09.820720 7775 consensus_queue.cc:1035] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:09.821170 6528 raft_consensus.cc:1261] T ae76780e32bb4e5cb4e1c86403544173 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:09.823930 7520 consensus_queue.cc:1035] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:09.826848 6791 raft_consensus.cc:1261] T 712bb27624184da8ba9ba0d1248abc02 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:09.827421 7758 consensus_queue.cc:1035] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:09.828660 6529 raft_consensus.cc:1261] T 712bb27624184da8ba9ba0d1248abc02 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:09.829044 7758 consensus_queue.cc:1035] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:09.837592 7806 mvcc.cc:205] Tried to move back new op lower bound from 7023594495253168128 to 7023594494290731008. Current Snapshot: MvccSnapshot[applied={T|T < 7023594495253168128}]
I20240503 13:59:09.851753 7820 mvcc.cc:205] Tried to move back new op lower bound from 7023594495282659328 to 7023594493679325184. Current Snapshot: MvccSnapshot[applied={T|T < 7023594495282659328}]
INSERT report
rows total: 2000
time total: 120.848 ms
time per row: 0.0604242 ms
I20240503 13:59:09.896031 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:09.901520 6770 tablet_service.cc:1467] Processing CreateTablet for tablet ea45c42d7fed4171a1b726b0684d2fa5 (DEFAULT_TABLE table=rebalance_test_table_5 [id=cb716c6127694172a7a76a19a23dc99e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:09.901733 6901 tablet_service.cc:1467] Processing CreateTablet for tablet ea45c42d7fed4171a1b726b0684d2fa5 (DEFAULT_TABLE table=rebalance_test_table_5 [id=cb716c6127694172a7a76a19a23dc99e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:09.901827 6770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea45c42d7fed4171a1b726b0684d2fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:09.901968 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea45c42d7fed4171a1b726b0684d2fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:09.903580 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 7203f103a97141dcb70f42c0d27faa37 (DEFAULT_TABLE table=rebalance_test_table_5 [id=cb716c6127694172a7a76a19a23dc99e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:09.903846 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7203f103a97141dcb70f42c0d27faa37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:09.904331 6640 tablet_service.cc:1467] Processing CreateTablet for tablet ea45c42d7fed4171a1b726b0684d2fa5 (DEFAULT_TABLE table=rebalance_test_table_5 [id=cb716c6127694172a7a76a19a23dc99e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:09.904563 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea45c42d7fed4171a1b726b0684d2fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:09.905627 7838 tablet_bootstrap.cc:492] T 7203f103a97141dcb70f42c0d27faa37 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:09.905750 6902 tablet_service.cc:1467] Processing CreateTablet for tablet 8f57f1ad5c604b5e957374bab029a5df (DEFAULT_TABLE table=rebalance_test_table_5 [id=cb716c6127694172a7a76a19a23dc99e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:09.905985 6902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f57f1ad5c604b5e957374bab029a5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:09.906545 7838 tablet_bootstrap.cc:654] T 7203f103a97141dcb70f42c0d27faa37 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:09.906847 7839 tablet_bootstrap.cc:492] T ea45c42d7fed4171a1b726b0684d2fa5 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:09.906935 6638 tablet_service.cc:1467] Processing CreateTablet for tablet 7203f103a97141dcb70f42c0d27faa37 (DEFAULT_TABLE table=rebalance_test_table_5 [id=cb716c6127694172a7a76a19a23dc99e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:09.907172 6638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7203f103a97141dcb70f42c0d27faa37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:09.907509 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 8f57f1ad5c604b5e957374bab029a5df (DEFAULT_TABLE table=rebalance_test_table_5 [id=cb716c6127694172a7a76a19a23dc99e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:09.907698 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f57f1ad5c604b5e957374bab029a5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:09.908399 7839 tablet_bootstrap.cc:654] T ea45c42d7fed4171a1b726b0684d2fa5 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:09.909407 7839 tablet_bootstrap.cc:492] T ea45c42d7fed4171a1b726b0684d2fa5 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:09.909484 7839 ts_tablet_manager.cc:1367] T ea45c42d7fed4171a1b726b0684d2fa5 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:59:09.909554 6770 tablet_service.cc:1467] Processing CreateTablet for tablet 8f57f1ad5c604b5e957374bab029a5df (DEFAULT_TABLE table=rebalance_test_table_5 [id=cb716c6127694172a7a76a19a23dc99e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:09.909584 6771 tablet_service.cc:1467] Processing CreateTablet for tablet 7203f103a97141dcb70f42c0d27faa37 (DEFAULT_TABLE table=rebalance_test_table_5 [id=cb716c6127694172a7a76a19a23dc99e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:09.909754 7837 tablet_bootstrap.cc:492] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:09.909817 6770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f57f1ad5c604b5e957374bab029a5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:09.909893 7839 raft_consensus.cc:348] T ea45c42d7fed4171a1b726b0684d2fa5 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:09.910029 7839 raft_consensus.cc:374] T ea45c42d7fed4171a1b726b0684d2fa5 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:09.910070 7839 raft_consensus.cc:729] T ea45c42d7fed4171a1b726b0684d2fa5 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:09.910200 7839 consensus_queue.cc:260] T ea45c42d7fed4171a1b726b0684d2fa5 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:09.910509 7839 ts_tablet_manager.cc:1398] T ea45c42d7fed4171a1b726b0684d2fa5 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:09.910691 7839 tablet_bootstrap.cc:492] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:09.910948 7837 tablet_bootstrap.cc:654] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:09.911588 7839 tablet_bootstrap.cc:654] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:09.911829 7836 tablet_bootstrap.cc:492] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:09.912182 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7203f103a97141dcb70f42c0d27faa37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:09.912580 7838 tablet_bootstrap.cc:492] T 7203f103a97141dcb70f42c0d27faa37 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:09.912671 7838 ts_tablet_manager.cc:1367] T 7203f103a97141dcb70f42c0d27faa37 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:59:09.912869 7836 tablet_bootstrap.cc:654] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:09.913179 7838 raft_consensus.cc:348] T 7203f103a97141dcb70f42c0d27faa37 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:09.913303 7838 raft_consensus.cc:374] T 7203f103a97141dcb70f42c0d27faa37 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:09.913350 7838 raft_consensus.cc:729] T 7203f103a97141dcb70f42c0d27faa37 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:09.913497 7838 consensus_queue.cc:260] T 7203f103a97141dcb70f42c0d27faa37 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:09.913782 7838 ts_tablet_manager.cc:1398] T 7203f103a97141dcb70f42c0d27faa37 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:09.915601 7837 tablet_bootstrap.cc:492] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:09.915699 7837 ts_tablet_manager.cc:1367] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 13:59:09.915735 7839 tablet_bootstrap.cc:492] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:09.915830 7839 ts_tablet_manager.cc:1367] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:09.916113 7837 raft_consensus.cc:348] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:09.916219 7837 raft_consensus.cc:374] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:09.916256 7837 raft_consensus.cc:729] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:09.916289 7839 raft_consensus.cc:348] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:09.916393 7839 raft_consensus.cc:374] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:09.916437 7839 raft_consensus.cc:729] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:09.916407 7837 consensus_queue.cc:260] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:09.916550 7839 consensus_queue.cc:260] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:09.916682 7836 tablet_bootstrap.cc:492] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:09.916752 7836 ts_tablet_manager.cc:1367] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:09.916831 7839 ts_tablet_manager.cc:1398] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:09.916989 7839 tablet_bootstrap.cc:492] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:09.917330 7837 ts_tablet_manager.cc:1398] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:09.917480 7837 tablet_bootstrap.cc:492] T 7203f103a97141dcb70f42c0d27faa37 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:09.917831 7839 tablet_bootstrap.cc:654] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:09.917757 7836 raft_consensus.cc:348] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:09.917872 7836 raft_consensus.cc:374] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:09.917901 7836 raft_consensus.cc:729] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:09.918056 7836 consensus_queue.cc:260] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:09.918453 7836 ts_tablet_manager.cc:1398] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:09.918624 7836 tablet_bootstrap.cc:492] T 8f57f1ad5c604b5e957374bab029a5df P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:09.918776 7837 tablet_bootstrap.cc:654] T 7203f103a97141dcb70f42c0d27faa37 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:09.919474 7836 tablet_bootstrap.cc:654] T 8f57f1ad5c604b5e957374bab029a5df P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:09.920696 7836 tablet_bootstrap.cc:492] T 8f57f1ad5c604b5e957374bab029a5df P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:09.920775 7836 ts_tablet_manager.cc:1367] T 8f57f1ad5c604b5e957374bab029a5df P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:09.921168 7836 raft_consensus.cc:348] T 8f57f1ad5c604b5e957374bab029a5df P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:09.921277 7836 raft_consensus.cc:374] T 8f57f1ad5c604b5e957374bab029a5df P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:09.921327 7839 tablet_bootstrap.cc:492] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:09.921325 7836 raft_consensus.cc:729] T 8f57f1ad5c604b5e957374bab029a5df P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:09.921403 7839 ts_tablet_manager.cc:1367] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 13:59:09.921435 7836 consensus_queue.cc:260] T 8f57f1ad5c604b5e957374bab029a5df P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:09.921705 7836 ts_tablet_manager.cc:1398] T 8f57f1ad5c604b5e957374bab029a5df P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:09.921818 7839 raft_consensus.cc:348] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:09.921916 7839 raft_consensus.cc:374] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:09.921952 7839 raft_consensus.cc:729] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:09.922075 7839 consensus_queue.cc:260] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:09.922868 7839 ts_tablet_manager.cc:1398] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 13:59:09.923589 7837 tablet_bootstrap.cc:492] T 7203f103a97141dcb70f42c0d27faa37 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:09.923677 7837 ts_tablet_manager.cc:1367] T 7203f103a97141dcb70f42c0d27faa37 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240503 13:59:09.924086 7837 raft_consensus.cc:348] T 7203f103a97141dcb70f42c0d27faa37 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:09.924180 7837 raft_consensus.cc:374] T 7203f103a97141dcb70f42c0d27faa37 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:09.924211 7837 raft_consensus.cc:729] T 7203f103a97141dcb70f42c0d27faa37 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:09.924331 7837 consensus_queue.cc:260] T 7203f103a97141dcb70f42c0d27faa37 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:09.924643 7837 ts_tablet_manager.cc:1398] T 7203f103a97141dcb70f42c0d27faa37 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:09.925302 7837 tablet_bootstrap.cc:492] T 8f57f1ad5c604b5e957374bab029a5df P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:09.926153 7837 tablet_bootstrap.cc:654] T 8f57f1ad5c604b5e957374bab029a5df P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:09.927253 7837 tablet_bootstrap.cc:492] T 8f57f1ad5c604b5e957374bab029a5df P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:09.927356 7837 ts_tablet_manager.cc:1367] T 8f57f1ad5c604b5e957374bab029a5df P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:09.927690 7837 raft_consensus.cc:348] T 8f57f1ad5c604b5e957374bab029a5df P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:09.927773 7837 raft_consensus.cc:374] T 8f57f1ad5c604b5e957374bab029a5df P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:09.927810 7837 raft_consensus.cc:729] T 8f57f1ad5c604b5e957374bab029a5df P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:09.927901 7837 consensus_queue.cc:260] T 8f57f1ad5c604b5e957374bab029a5df P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:09.928145 7837 ts_tablet_manager.cc:1398] T 8f57f1ad5c604b5e957374bab029a5df P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:09.949352 7828 raft_consensus.cc:482] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:09.949486 7828 raft_consensus.cc:504] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:09.949901 7828 leader_election.cc:288] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:09.950294 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f57f1ad5c604b5e957374bab029a5df" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:09.950451 6922 raft_consensus.cc:2454] T 8f57f1ad5c604b5e957374bab029a5df P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:09.950708 6595 leader_election.cc:302] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:09.950340 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f57f1ad5c604b5e957374bab029a5df" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:09.950881 7828 raft_consensus.cc:2781] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:09.950927 7828 raft_consensus.cc:482] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:09.950960 7828 raft_consensus.cc:3037] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.950985 6791 raft_consensus.cc:2454] T 8f57f1ad5c604b5e957374bab029a5df P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:09.951856 7828 raft_consensus.cc:504] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:09.952181 7828 leader_election.cc:288] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:09.952461 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f57f1ad5c604b5e957374bab029a5df" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:09.952481 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f57f1ad5c604b5e957374bab029a5df" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:09.952584 6922 raft_consensus.cc:3037] T 8f57f1ad5c604b5e957374bab029a5df P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.952610 6791 raft_consensus.cc:3037] T 8f57f1ad5c604b5e957374bab029a5df P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.953512 6922 raft_consensus.cc:2454] T 8f57f1ad5c604b5e957374bab029a5df P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:09.953512 6791 raft_consensus.cc:2454] T 8f57f1ad5c604b5e957374bab029a5df P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:09.953817 6595 leader_election.cc:302] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:09.953969 7828 raft_consensus.cc:2781] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:09.954044 7828 raft_consensus.cc:686] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 13:59:09.954201 7828 consensus_queue.cc:237] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:09.955850 6379 catalog_manager.cc:5484] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:09.959038 7834 raft_consensus.cc:482] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:09.959136 7834 raft_consensus.cc:504] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:09.959522 7834 leader_election.cc:288] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:09.959848 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea45c42d7fed4171a1b726b0684d2fa5" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:09.959983 6660 raft_consensus.cc:2454] T ea45c42d7fed4171a1b726b0684d2fa5 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:09.960003 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea45c42d7fed4171a1b726b0684d2fa5" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:09.960120 6922 raft_consensus.cc:2454] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:09.960271 6727 leader_election.cc:302] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:09.960440 7834 raft_consensus.cc:2781] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:09.960489 7834 raft_consensus.cc:482] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:09.960526 7834 raft_consensus.cc:3037] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.961336 7834 raft_consensus.cc:504] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:09.961613 7834 leader_election.cc:288] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:09.961958 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea45c42d7fed4171a1b726b0684d2fa5" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:09.962070 6922 raft_consensus.cc:3037] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.962080 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea45c42d7fed4171a1b726b0684d2fa5" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:09.962169 6660 raft_consensus.cc:3037] T ea45c42d7fed4171a1b726b0684d2fa5 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.962899 6922 raft_consensus.cc:2454] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:09.963140 6726 leader_election.cc:302] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:09.963336 7834 raft_consensus.cc:2781] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:09.963443 7834 raft_consensus.cc:686] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:09.963572 7834 consensus_queue.cc:237] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:09.965123 6379 catalog_manager.cc:5484] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:09.983986 6660 raft_consensus.cc:2454] T ea45c42d7fed4171a1b726b0684d2fa5 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:10.040717 7828 raft_consensus.cc:482] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:10.040874 7828 raft_consensus.cc:504] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:10.041275 7828 leader_election.cc:288] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:10.041615 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7203f103a97141dcb70f42c0d27faa37" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:10.041639 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7203f103a97141dcb70f42c0d27faa37" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:10.041774 6791 raft_consensus.cc:2454] T 7203f103a97141dcb70f42c0d27faa37 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:10.041780 6528 raft_consensus.cc:2454] T 7203f103a97141dcb70f42c0d27faa37 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:10.042049 6594 leader_election.cc:302] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:10.042213 7828 raft_consensus.cc:2781] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:10.042276 7828 raft_consensus.cc:482] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:10.042308 7828 raft_consensus.cc:3037] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:10.043133 7828 raft_consensus.cc:504] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:10.043474 7828 leader_election.cc:288] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:10.043804 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7203f103a97141dcb70f42c0d27faa37" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:10.043809 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7203f103a97141dcb70f42c0d27faa37" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:10.043921 6528 raft_consensus.cc:3037] T 7203f103a97141dcb70f42c0d27faa37 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:10.043931 6791 raft_consensus.cc:3037] T 7203f103a97141dcb70f42c0d27faa37 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:10.044677 6528 raft_consensus.cc:2454] T 7203f103a97141dcb70f42c0d27faa37 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:10.044726 6791 raft_consensus.cc:2454] T 7203f103a97141dcb70f42c0d27faa37 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:10.044971 6595 leader_election.cc:302] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:10.045181 7828 raft_consensus.cc:2781] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:10.045274 7828 raft_consensus.cc:686] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 13:59:10.045455 7828 consensus_queue.cc:237] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:10.047096 6379 catalog_manager.cc:5484] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } }
W20240503 13:59:10.164882 7853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:10.165089 7853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:10.168005 7853 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 13:59:10.253142 6789 raft_consensus.cc:1261] T 7203f103a97141dcb70f42c0d27faa37 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:10.253147 6791 raft_consensus.cc:1261] T 8f57f1ad5c604b5e957374bab029a5df P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:10.253587 6922 raft_consensus.cc:1261] T 8f57f1ad5c604b5e957374bab029a5df P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:10.253571 7828 consensus_queue.cc:1035] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:10.253929 7828 consensus_queue.cc:1035] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:10.254896 6660 raft_consensus.cc:1261] T ea45c42d7fed4171a1b726b0684d2fa5 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:10.255358 7574 consensus_queue.cc:1035] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:10.255517 6922 raft_consensus.cc:1261] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:10.256034 7775 consensus_queue.cc:1035] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:10.258013 7834 consensus_queue.cc:1035] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:10.262598 7809 mvcc.cc:205] Tried to move back new op lower bound from 7023594497030172672 to 7023594495813873664. Current Snapshot: MvccSnapshot[applied={T|T < 7023594497030172672}]
I20240503 13:59:10.267971 6528 raft_consensus.cc:1261] T 7203f103a97141dcb70f42c0d27faa37 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:10.269330 7758 consensus_queue.cc:1035] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:10.280561 7818 mvcc.cc:205] Tried to move back new op lower bound from 7023594497030496256 to 7023594496187334656. Current Snapshot: MvccSnapshot[applied={T|T < 7023594497030496256}]
INSERT report
rows total: 2000
time total: 109.796 ms
time per row: 0.0548978 ms
I20240503 13:59:10.317030 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:10.322270 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 927176cab28741f882a2021868e3d744 (DEFAULT_TABLE table=rebalance_test_table_6 [id=db76abc34823418f9b242b9c4575cb62]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:10.322412 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 927176cab28741f882a2021868e3d744 (DEFAULT_TABLE table=rebalance_test_table_6 [id=db76abc34823418f9b242b9c4575cb62]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:10.322574 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 927176cab28741f882a2021868e3d744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:10.322693 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 927176cab28741f882a2021868e3d744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:10.323168 6638 tablet_service.cc:1467] Processing CreateTablet for tablet 5e7500549f5c49b1b1e8ab1781bfab4b (DEFAULT_TABLE table=rebalance_test_table_6 [id=db76abc34823418f9b242b9c4575cb62]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:10.323462 6638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e7500549f5c49b1b1e8ab1781bfab4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:10.323635 6902 tablet_service.cc:1467] Processing CreateTablet for tablet 927176cab28741f882a2021868e3d744 (DEFAULT_TABLE table=rebalance_test_table_6 [id=db76abc34823418f9b242b9c4575cb62]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:10.323889 6902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 927176cab28741f882a2021868e3d744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:10.324311 6901 tablet_service.cc:1467] Processing CreateTablet for tablet 76799d8d81324f898705ee572782261c (DEFAULT_TABLE table=rebalance_test_table_6 [id=db76abc34823418f9b242b9c4575cb62]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:10.324381 6900 tablet_service.cc:1467] Processing CreateTablet for tablet 5e7500549f5c49b1b1e8ab1781bfab4b (DEFAULT_TABLE table=rebalance_test_table_6 [id=db76abc34823418f9b242b9c4575cb62]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:10.324550 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76799d8d81324f898705ee572782261c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:10.324877 6900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e7500549f5c49b1b1e8ab1781bfab4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:10.325116 7839 tablet_bootstrap.cc:492] T 927176cab28741f882a2021868e3d744 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:10.325654 6770 tablet_service.cc:1467] Processing CreateTablet for tablet 76799d8d81324f898705ee572782261c (DEFAULT_TABLE table=rebalance_test_table_6 [id=db76abc34823418f9b242b9c4575cb62]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:10.325781 6509 tablet_service.cc:1467] Processing CreateTablet for tablet 76799d8d81324f898705ee572782261c (DEFAULT_TABLE table=rebalance_test_table_6 [id=db76abc34823418f9b242b9c4575cb62]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:10.325925 7839 tablet_bootstrap.cc:654] T 927176cab28741f882a2021868e3d744 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:10.325925 6770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76799d8d81324f898705ee572782261c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:10.326020 6509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76799d8d81324f898705ee572782261c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:10.326383 6507 tablet_service.cc:1467] Processing CreateTablet for tablet 5e7500549f5c49b1b1e8ab1781bfab4b (DEFAULT_TABLE table=rebalance_test_table_6 [id=db76abc34823418f9b242b9c4575cb62]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:10.326607 6507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e7500549f5c49b1b1e8ab1781bfab4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:10.326920 7839 tablet_bootstrap.cc:492] T 927176cab28741f882a2021868e3d744 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:10.327005 7839 ts_tablet_manager.cc:1367] T 927176cab28741f882a2021868e3d744 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 13:59:10.327448 7839 raft_consensus.cc:348] T 927176cab28741f882a2021868e3d744 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:10.327551 7839 raft_consensus.cc:374] T 927176cab28741f882a2021868e3d744 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:10.327580 7839 raft_consensus.cc:729] T 927176cab28741f882a2021868e3d744 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:10.327697 7839 consensus_queue.cc:260] T 927176cab28741f882a2021868e3d744 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:10.328006 7839 ts_tablet_manager.cc:1398] T 927176cab28741f882a2021868e3d744 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:10.328047 7837 tablet_bootstrap.cc:492] T 76799d8d81324f898705ee572782261c P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:10.328177 7839 tablet_bootstrap.cc:492] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:10.328761 7837 tablet_bootstrap.cc:654] T 76799d8d81324f898705ee572782261c P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:10.328992 7839 tablet_bootstrap.cc:654] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:10.329788 7837 tablet_bootstrap.cc:492] T 76799d8d81324f898705ee572782261c P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:10.329855 7837 ts_tablet_manager.cc:1367] T 76799d8d81324f898705ee572782261c P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:10.330168 7837 raft_consensus.cc:348] T 76799d8d81324f898705ee572782261c P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:10.330278 7837 raft_consensus.cc:374] T 76799d8d81324f898705ee572782261c P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:10.330323 7837 raft_consensus.cc:729] T 76799d8d81324f898705ee572782261c P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:10.330448 7837 consensus_queue.cc:260] T 76799d8d81324f898705ee572782261c P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:10.330718 7837 ts_tablet_manager.cc:1398] T 76799d8d81324f898705ee572782261c P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:10.331269 7839 tablet_bootstrap.cc:492] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:10.331399 7839 ts_tablet_manager.cc:1367] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:10.331790 7839 raft_consensus.cc:348] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:10.331881 7839 raft_consensus.cc:374] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:10.331924 7839 raft_consensus.cc:729] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:10.332391 7839 consensus_queue.cc:260] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:10.332700 7839 ts_tablet_manager.cc:1398] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:10.349097 7836 tablet_bootstrap.cc:492] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:10.349205 7838 tablet_bootstrap.cc:492] T 927176cab28741f882a2021868e3d744 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:10.349787 7836 tablet_bootstrap.cc:654] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:10.350037 7838 tablet_bootstrap.cc:654] T 927176cab28741f882a2021868e3d744 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:10.350952 7838 tablet_bootstrap.cc:492] T 927176cab28741f882a2021868e3d744 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:10.351044 7838 ts_tablet_manager.cc:1367] T 927176cab28741f882a2021868e3d744 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:10.351197 7836 tablet_bootstrap.cc:492] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:10.351514 7838 raft_consensus.cc:348] T 927176cab28741f882a2021868e3d744 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:10.351621 7838 raft_consensus.cc:374] T 927176cab28741f882a2021868e3d744 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:10.351656 7838 raft_consensus.cc:729] T 927176cab28741f882a2021868e3d744 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:10.351298 7836 ts_tablet_manager.cc:1367] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:10.351764 7838 consensus_queue.cc:260] T 927176cab28741f882a2021868e3d744 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:10.352047 7838 ts_tablet_manager.cc:1398] T 927176cab28741f882a2021868e3d744 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:10.352197 7838 tablet_bootstrap.cc:492] T 5e7500549f5c49b1b1e8ab1781bfab4b P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:10.352154 7836 raft_consensus.cc:348] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:10.352246 7836 raft_consensus.cc:374] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:10.352283 7836 raft_consensus.cc:729] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:10.352423 7836 consensus_queue.cc:260] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:10.352672 7836 ts_tablet_manager.cc:1398] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:10.352825 7836 tablet_bootstrap.cc:492] T 5e7500549f5c49b1b1e8ab1781bfab4b P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:10.353039 7838 tablet_bootstrap.cc:654] T 5e7500549f5c49b1b1e8ab1781bfab4b P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:10.353673 7836 tablet_bootstrap.cc:654] T 5e7500549f5c49b1b1e8ab1781bfab4b P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:10.354148 7838 tablet_bootstrap.cc:492] T 5e7500549f5c49b1b1e8ab1781bfab4b P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:10.354233 7838 ts_tablet_manager.cc:1367] T 5e7500549f5c49b1b1e8ab1781bfab4b P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:10.354882 7838 raft_consensus.cc:348] T 5e7500549f5c49b1b1e8ab1781bfab4b P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:10.354950 7836 tablet_bootstrap.cc:492] T 5e7500549f5c49b1b1e8ab1781bfab4b P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:10.354969 7838 raft_consensus.cc:374] T 5e7500549f5c49b1b1e8ab1781bfab4b P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:10.354991 7838 raft_consensus.cc:729] T 5e7500549f5c49b1b1e8ab1781bfab4b P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:10.355022 7836 ts_tablet_manager.cc:1367] T 5e7500549f5c49b1b1e8ab1781bfab4b P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:10.355115 7838 consensus_queue.cc:260] T 5e7500549f5c49b1b1e8ab1781bfab4b P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:10.355374 7838 ts_tablet_manager.cc:1398] T 5e7500549f5c49b1b1e8ab1781bfab4b P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:10.355463 7836 raft_consensus.cc:348] T 5e7500549f5c49b1b1e8ab1781bfab4b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:10.355582 7836 raft_consensus.cc:374] T 5e7500549f5c49b1b1e8ab1781bfab4b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:10.355618 7836 raft_consensus.cc:729] T 5e7500549f5c49b1b1e8ab1781bfab4b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:10.355643 7838 tablet_bootstrap.cc:492] T 76799d8d81324f898705ee572782261c P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:10.355760 7836 consensus_queue.cc:260] T 5e7500549f5c49b1b1e8ab1781bfab4b P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:10.356117 7836 ts_tablet_manager.cc:1398] T 5e7500549f5c49b1b1e8ab1781bfab4b P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:10.356290 7836 tablet_bootstrap.cc:492] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:10.356513 7838 tablet_bootstrap.cc:654] T 76799d8d81324f898705ee572782261c P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:10.357120 7836 tablet_bootstrap.cc:654] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:10.358197 7836 tablet_bootstrap.cc:492] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:10.358237 7838 tablet_bootstrap.cc:492] T 76799d8d81324f898705ee572782261c P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:10.358274 7836 ts_tablet_manager.cc:1367] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:10.358309 7838 ts_tablet_manager.cc:1367] T 76799d8d81324f898705ee572782261c P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:10.358755 7836 raft_consensus.cc:348] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:10.358878 7836 raft_consensus.cc:374] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:10.358908 7836 raft_consensus.cc:729] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:10.358991 7838 raft_consensus.cc:348] T 76799d8d81324f898705ee572782261c P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:10.359040 7836 consensus_queue.cc:260] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:10.359102 7838 raft_consensus.cc:374] T 76799d8d81324f898705ee572782261c P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:10.359329 7836 ts_tablet_manager.cc:1398] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:10.359403 7838 raft_consensus.cc:729] T 76799d8d81324f898705ee572782261c P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:10.359563 7838 consensus_queue.cc:260] T 76799d8d81324f898705ee572782261c P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:10.359854 7838 ts_tablet_manager.cc:1398] T 76799d8d81324f898705ee572782261c P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:10.442896 7885 raft_consensus.cc:482] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:10.443018 7885 raft_consensus.cc:504] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:10.443485 7885 leader_election.cc:288] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:10.443783 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e7500549f5c49b1b1e8ab1781bfab4b" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:10.443934 6528 raft_consensus.cc:2454] T 5e7500549f5c49b1b1e8ab1781bfab4b P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:10.443907 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e7500549f5c49b1b1e8ab1781bfab4b" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:10.444032 6922 raft_consensus.cc:2454] T 5e7500549f5c49b1b1e8ab1781bfab4b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:10.444234 6595 leader_election.cc:302] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:10.444407 7885 raft_consensus.cc:2781] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:10.444466 7885 raft_consensus.cc:482] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:10.444517 7885 raft_consensus.cc:3037] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:10.445355 7885 raft_consensus.cc:504] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:10.445657 7885 leader_election.cc:288] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:10.445960 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e7500549f5c49b1b1e8ab1781bfab4b" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:10.446071 6528 raft_consensus.cc:3037] T 5e7500549f5c49b1b1e8ab1781bfab4b P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:10.446022 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e7500549f5c49b1b1e8ab1781bfab4b" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:10.446156 6922 raft_consensus.cc:3037] T 5e7500549f5c49b1b1e8ab1781bfab4b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:10.446866 6528 raft_consensus.cc:2454] T 5e7500549f5c49b1b1e8ab1781bfab4b P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:10.446998 6922 raft_consensus.cc:2454] T 5e7500549f5c49b1b1e8ab1781bfab4b P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:10.447160 6595 leader_election.cc:302] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:10.447369 7885 raft_consensus.cc:2781] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:10.447532 7885 raft_consensus.cc:686] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 13:59:10.447666 7885 consensus_queue.cc:237] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:10.449440 6380 catalog_manager.cc:5484] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:10.517724 7844 raft_consensus.cc:482] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:10.517843 7844 raft_consensus.cc:504] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:10.518219 7844 leader_election.cc:288] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:10.518627 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76799d8d81324f898705ee572782261c" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:10.518646 6789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76799d8d81324f898705ee572782261c" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:10.518774 6528 raft_consensus.cc:2454] T 76799d8d81324f898705ee572782261c P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:10.518803 6789 raft_consensus.cc:2454] T 76799d8d81324f898705ee572782261c P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:10.519078 6857 leader_election.cc:302] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:10.519243 7844 raft_consensus.cc:2781] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:10.519325 7844 raft_consensus.cc:482] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:10.519361 7844 raft_consensus.cc:3037] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:10.520165 7844 raft_consensus.cc:504] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:10.520481 7844 leader_election.cc:288] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:10.520821 6789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76799d8d81324f898705ee572782261c" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:10.520856 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76799d8d81324f898705ee572782261c" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:10.520954 6528 raft_consensus.cc:3037] T 76799d8d81324f898705ee572782261c P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:10.520951 6789 raft_consensus.cc:3037] T 76799d8d81324f898705ee572782261c P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:10.521653 6789 raft_consensus.cc:2454] T 76799d8d81324f898705ee572782261c P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:10.521678 6528 raft_consensus.cc:2454] T 76799d8d81324f898705ee572782261c P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:10.521924 6856 leader_election.cc:302] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:10.522070 7844 raft_consensus.cc:2781] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:10.522158 7844 raft_consensus.cc:686] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:10.522321 7844 consensus_queue.cc:237] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:10.524144 6380 catalog_manager.cc:5484] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:10.692842 7844 raft_consensus.cc:482] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:10.692970 7844 raft_consensus.cc:504] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:10.693375 7844 leader_election.cc:288] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:10.693733 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "927176cab28741f882a2021868e3d744" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:10.693743 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "927176cab28741f882a2021868e3d744" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:10.693863 6528 raft_consensus.cc:2454] T 927176cab28741f882a2021868e3d744 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:10.693881 6660 raft_consensus.cc:2454] T 927176cab28741f882a2021868e3d744 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:10.694149 6857 leader_election.cc:302] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:10.694319 7844 raft_consensus.cc:2781] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:10.694391 7844 raft_consensus.cc:482] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:10.694430 7844 raft_consensus.cc:3037] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:10.695262 7844 raft_consensus.cc:504] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:10.695618 7844 leader_election.cc:288] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:10.695904 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "927176cab28741f882a2021868e3d744" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:10.695932 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "927176cab28741f882a2021868e3d744" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:10.696019 6528 raft_consensus.cc:3037] T 927176cab28741f882a2021868e3d744 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:10.696046 6660 raft_consensus.cc:3037] T 927176cab28741f882a2021868e3d744 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:10.696805 6528 raft_consensus.cc:2454] T 927176cab28741f882a2021868e3d744 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:10.696808 6660 raft_consensus.cc:2454] T 927176cab28741f882a2021868e3d744 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:10.697049 6857 leader_election.cc:302] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:10.697242 7844 raft_consensus.cc:2781] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:10.697321 7844 raft_consensus.cc:686] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:10.697448 7844 consensus_queue.cc:237] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:10.699060 6380 catalog_manager.cc:5484] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:10.844245 7758 consensus_queue.cc:1035] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:10.848017 7758 consensus_queue.cc:1035] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 13:59:10.886906 7905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:10.887094 7905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:10.889953 7905 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 13:59:10.931672 7844 consensus_queue.cc:1035] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:10.945511 7535 consensus_queue.cc:1035] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:10.984050 6660 raft_consensus.cc:1261] T 927176cab28741f882a2021868e3d744 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:10.984483 7844 consensus_queue.cc:1035] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:10.987921 6528 raft_consensus.cc:1261] T 927176cab28741f882a2021868e3d744 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:10.988301 7844 consensus_queue.cc:1035] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 126.463 ms
time per row: 0.0632315 ms
I20240503 13:59:11.055945 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:11.060894 6770 tablet_service.cc:1467] Processing CreateTablet for tablet ce9410a4ed09491b8d37456ae44880a1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=73b39055500b4618821faa094d33670a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:11.060961 6509 tablet_service.cc:1467] Processing CreateTablet for tablet ce9410a4ed09491b8d37456ae44880a1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=73b39055500b4618821faa094d33670a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:11.061205 6770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce9410a4ed09491b8d37456ae44880a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:11.061228 6509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce9410a4ed09491b8d37456ae44880a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:11.061234 6638 tablet_service.cc:1467] Processing CreateTablet for tablet ce9410a4ed09491b8d37456ae44880a1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=73b39055500b4618821faa094d33670a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:11.061476 6638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce9410a4ed09491b8d37456ae44880a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:11.062209 6640 tablet_service.cc:1467] Processing CreateTablet for tablet db14294a5205460dbdd0f7588d92c1b5 (DEFAULT_TABLE table=rebalance_test_table_7 [id=73b39055500b4618821faa094d33670a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:11.062439 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db14294a5205460dbdd0f7588d92c1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:11.063618 7965 tablet_bootstrap.cc:492] T ce9410a4ed09491b8d37456ae44880a1 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:11.063773 7964 tablet_bootstrap.cc:492] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:11.063972 6770 tablet_service.cc:1467] Processing CreateTablet for tablet d55d270456d3460586ff3cd41787b5d2 (DEFAULT_TABLE table=rebalance_test_table_7 [id=73b39055500b4618821faa094d33670a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:11.064168 6770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55d270456d3460586ff3cd41787b5d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:11.064464 7965 tablet_bootstrap.cc:654] T ce9410a4ed09491b8d37456ae44880a1 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:11.064735 7964 tablet_bootstrap.cc:654] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:11.065156 6771 tablet_service.cc:1467] Processing CreateTablet for tablet db14294a5205460dbdd0f7588d92c1b5 (DEFAULT_TABLE table=rebalance_test_table_7 [id=73b39055500b4618821faa094d33670a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:11.065375 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db14294a5205460dbdd0f7588d92c1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:11.065877 7964 tablet_bootstrap.cc:492] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:11.065979 7964 ts_tablet_manager.cc:1367] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:11.066432 7964 raft_consensus.cc:348] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:11.066524 7964 raft_consensus.cc:374] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:11.066558 7964 raft_consensus.cc:729] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:11.066676 7965 tablet_bootstrap.cc:492] T ce9410a4ed09491b8d37456ae44880a1 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:11.066699 7964 consensus_queue.cc:260] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:11.066774 7965 ts_tablet_manager.cc:1367] T ce9410a4ed09491b8d37456ae44880a1 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:11.067214 7965 raft_consensus.cc:348] T ce9410a4ed09491b8d37456ae44880a1 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:11.067425 7965 raft_consensus.cc:374] T ce9410a4ed09491b8d37456ae44880a1 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:11.067468 7965 raft_consensus.cc:729] T ce9410a4ed09491b8d37456ae44880a1 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:11.067620 7964 ts_tablet_manager.cc:1398] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:11.067589 7965 consensus_queue.cc:260] T ce9410a4ed09491b8d37456ae44880a1 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:11.067674 7963 tablet_bootstrap.cc:492] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:11.067759 7964 tablet_bootstrap.cc:492] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:11.067842 7965 ts_tablet_manager.cc:1398] T ce9410a4ed09491b8d37456ae44880a1 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:11.067826 6901 tablet_service.cc:1467] Processing CreateTablet for tablet d55d270456d3460586ff3cd41787b5d2 (DEFAULT_TABLE table=rebalance_test_table_7 [id=73b39055500b4618821faa094d33670a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:11.067983 7965 tablet_bootstrap.cc:492] T db14294a5205460dbdd0f7588d92c1b5 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:11.068075 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55d270456d3460586ff3cd41787b5d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:11.068506 7963 tablet_bootstrap.cc:654] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:11.068529 7964 tablet_bootstrap.cc:654] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:11.068797 7965 tablet_bootstrap.cc:654] T db14294a5205460dbdd0f7588d92c1b5 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:11.069339 7520 raft_consensus.cc:482] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:11.069411 7520 raft_consensus.cc:504] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:11.069746 7520 leader_election.cc:288] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:11.069919 6900 tablet_service.cc:1467] Processing CreateTablet for tablet db14294a5205460dbdd0f7588d92c1b5 (DEFAULT_TABLE table=rebalance_test_table_7 [id=73b39055500b4618821faa094d33670a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:11.070134 6900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db14294a5205460dbdd0f7588d92c1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:11.070466 7963 tablet_bootstrap.cc:492] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:11.070544 7963 ts_tablet_manager.cc:1367] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:11.070583 7971 tablet_bootstrap.cc:492] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:11.070924 7963 raft_consensus.cc:348] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:11.071035 7963 raft_consensus.cc:374] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:11.071063 7963 raft_consensus.cc:729] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:11.071169 7963 consensus_queue.cc:260] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:11.071413 7971 tablet_bootstrap.cc:654] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:11.071462 7963 ts_tablet_manager.cc:1398] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:11.072156 6509 tablet_service.cc:1467] Processing CreateTablet for tablet d55d270456d3460586ff3cd41787b5d2 (DEFAULT_TABLE table=rebalance_test_table_7 [id=73b39055500b4618821faa094d33670a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:11.072196 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce9410a4ed09491b8d37456ae44880a1" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:11.072297 6528 raft_consensus.cc:2454] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:11.072368 6509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55d270456d3460586ff3cd41787b5d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:11.072458 7971 tablet_bootstrap.cc:492] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:11.072515 7971 ts_tablet_manager.cc:1367] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:11.072793 7971 raft_consensus.cc:348] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:11.072854 7971 raft_consensus.cc:374] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:11.072871 7971 raft_consensus.cc:729] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:11.072965 7971 consensus_queue.cc:260] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:11.073195 7971 ts_tablet_manager.cc:1398] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:11.073302 7971 tablet_bootstrap.cc:492] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:11.073644 6726 leader_election.cc:302] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:11.073897 7936 raft_consensus.cc:2781] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:11.073940 7936 raft_consensus.cc:482] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:11.073951 7971 tablet_bootstrap.cc:654] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:11.073976 7936 raft_consensus.cc:3037] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:11.074096 7963 tablet_bootstrap.cc:492] T d55d270456d3460586ff3cd41787b5d2 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:11.074527 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce9410a4ed09491b8d37456ae44880a1" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:11.074627 6660 raft_consensus.cc:2454] T ce9410a4ed09491b8d37456ae44880a1 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:11.074796 7936 raft_consensus.cc:504] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:11.074957 7963 tablet_bootstrap.cc:654] T d55d270456d3460586ff3cd41787b5d2 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:11.075093 7936 leader_election.cc:288] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:11.075320 7971 tablet_bootstrap.cc:492] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:11.075390 7971 ts_tablet_manager.cc:1367] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:11.075795 7971 raft_consensus.cc:348] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:11.075893 7971 raft_consensus.cc:374] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:11.075930 7971 raft_consensus.cc:729] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:11.076035 7971 consensus_queue.cc:260] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:11.076273 7971 ts_tablet_manager.cc:1398] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:11.077061 7964 tablet_bootstrap.cc:492] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:11.077144 7964 ts_tablet_manager.cc:1367] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 13:59:11.077170 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce9410a4ed09491b8d37456ae44880a1" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:11.077260 6660 raft_consensus.cc:3037] T ce9410a4ed09491b8d37456ae44880a1 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:11.077559 7964 raft_consensus.cc:348] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:11.077646 7964 raft_consensus.cc:374] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:11.077677 7964 raft_consensus.cc:729] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:11.077724 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce9410a4ed09491b8d37456ae44880a1" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:11.077771 7964 consensus_queue.cc:260] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:11.077817 6528 raft_consensus.cc:3037] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:11.078059 7964 ts_tablet_manager.cc:1398] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:11.078135 6660 raft_consensus.cc:2454] T ce9410a4ed09491b8d37456ae44880a1 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:11.078217 7964 tablet_bootstrap.cc:492] T db14294a5205460dbdd0f7588d92c1b5 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:11.078461 6727 leader_election.cc:302] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:11.078634 7936 raft_consensus.cc:2781] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:11.078704 7936 raft_consensus.cc:686] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:11.078775 6528 raft_consensus.cc:2454] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:11.078804 7936 consensus_queue.cc:237] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:11.079018 7964 tablet_bootstrap.cc:654] T db14294a5205460dbdd0f7588d92c1b5 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:11.079900 7965 tablet_bootstrap.cc:492] T db14294a5205460dbdd0f7588d92c1b5 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:11.079981 7965 ts_tablet_manager.cc:1367] T db14294a5205460dbdd0f7588d92c1b5 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240503 13:59:11.080224 7964 tablet_bootstrap.cc:492] T db14294a5205460dbdd0f7588d92c1b5 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:11.080296 7964 ts_tablet_manager.cc:1367] T db14294a5205460dbdd0f7588d92c1b5 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:11.080380 7965 raft_consensus.cc:348] T db14294a5205460dbdd0f7588d92c1b5 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:11.080520 7965 raft_consensus.cc:374] T db14294a5205460dbdd0f7588d92c1b5 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:11.080562 7965 raft_consensus.cc:729] T db14294a5205460dbdd0f7588d92c1b5 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:11.080679 7965 consensus_queue.cc:260] T db14294a5205460dbdd0f7588d92c1b5 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:11.080698 7964 raft_consensus.cc:348] T db14294a5205460dbdd0f7588d92c1b5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:11.080775 7964 raft_consensus.cc:374] T db14294a5205460dbdd0f7588d92c1b5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:11.080812 7964 raft_consensus.cc:729] T db14294a5205460dbdd0f7588d92c1b5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:11.080973 7965 ts_tablet_manager.cc:1398] T db14294a5205460dbdd0f7588d92c1b5 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:11.081305 7964 consensus_queue.cc:260] T db14294a5205460dbdd0f7588d92c1b5 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:11.081681 7964 ts_tablet_manager.cc:1398] T db14294a5205460dbdd0f7588d92c1b5 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:11.082177 7963 tablet_bootstrap.cc:492] T d55d270456d3460586ff3cd41787b5d2 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:11.082383 7963 ts_tablet_manager.cc:1367] T d55d270456d3460586ff3cd41787b5d2 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:59:11.082895 7963 raft_consensus.cc:348] T d55d270456d3460586ff3cd41787b5d2 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:11.083119 7963 raft_consensus.cc:374] T d55d270456d3460586ff3cd41787b5d2 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:11.083161 7963 raft_consensus.cc:729] T d55d270456d3460586ff3cd41787b5d2 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:11.083305 7963 consensus_queue.cc:260] T d55d270456d3460586ff3cd41787b5d2 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:11.083674 7963 ts_tablet_manager.cc:1398] T d55d270456d3460586ff3cd41787b5d2 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:11.083667 6382 catalog_manager.cc:5484] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:11.108011 7520 raft_consensus.cc:482] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:11.108125 7520 raft_consensus.cc:504] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:11.108521 7520 leader_election.cc:288] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:11.108887 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55d270456d3460586ff3cd41787b5d2" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:11.108947 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55d270456d3460586ff3cd41787b5d2" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:11.109031 6922 raft_consensus.cc:2454] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:11.109059 6528 raft_consensus.cc:2454] T d55d270456d3460586ff3cd41787b5d2 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:11.109320 6726 leader_election.cc:302] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:11.109504 7520 raft_consensus.cc:2781] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:11.109565 7520 raft_consensus.cc:482] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:11.109589 7520 raft_consensus.cc:3037] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:11.110334 7520 raft_consensus.cc:504] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:11.110625 7520 leader_election.cc:288] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:11.110950 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55d270456d3460586ff3cd41787b5d2" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:11.110989 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55d270456d3460586ff3cd41787b5d2" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:11.111070 6528 raft_consensus.cc:3037] T d55d270456d3460586ff3cd41787b5d2 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:11.111074 6922 raft_consensus.cc:3037] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:11.111800 6528 raft_consensus.cc:2454] T d55d270456d3460586ff3cd41787b5d2 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:11.111879 6922 raft_consensus.cc:2454] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:11.112061 6726 leader_election.cc:302] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:11.112241 7520 raft_consensus.cc:2781] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:11.112332 7520 raft_consensus.cc:686] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:11.112485 7520 consensus_queue.cc:237] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:11.114114 6383 catalog_manager.cc:5484] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:11.253572 7961 raft_consensus.cc:482] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:11.253731 7961 raft_consensus.cc:504] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:11.254192 7961 leader_election.cc:288] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:11.254750 6789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db14294a5205460dbdd0f7588d92c1b5" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:11.254809 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db14294a5205460dbdd0f7588d92c1b5" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:11.254994 6660 raft_consensus.cc:2454] T db14294a5205460dbdd0f7588d92c1b5 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:11.254901 6789 raft_consensus.cc:2454] T db14294a5205460dbdd0f7588d92c1b5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:11.255336 6858 leader_election.cc:302] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:11.255532 7961 raft_consensus.cc:2781] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:11.255599 7961 raft_consensus.cc:482] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:11.255625 7961 raft_consensus.cc:3037] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:11.256492 7961 raft_consensus.cc:504] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:11.256809 7961 leader_election.cc:288] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:11.257151 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db14294a5205460dbdd0f7588d92c1b5" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:11.257149 6789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db14294a5205460dbdd0f7588d92c1b5" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:11.257253 6660 raft_consensus.cc:3037] T db14294a5205460dbdd0f7588d92c1b5 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:11.257262 6789 raft_consensus.cc:3037] T db14294a5205460dbdd0f7588d92c1b5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:11.258037 6789 raft_consensus.cc:2454] T db14294a5205460dbdd0f7588d92c1b5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:11.258138 6660 raft_consensus.cc:2454] T db14294a5205460dbdd0f7588d92c1b5 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:11.258348 6856 leader_election.cc:302] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:11.258575 7961 raft_consensus.cc:2781] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:11.258652 7961 raft_consensus.cc:686] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:11.258798 7961 consensus_queue.cc:237] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:11.260875 6383 catalog_manager.cc:5484] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:11.414683 7979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:11.414884 7979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:11.417764 7979 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 13:59:11.483687 7520 consensus_queue.cc:1035] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:11.490002 7520 consensus_queue.cc:1035] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:11.503204 6528 raft_consensus.cc:1261] T d55d270456d3460586ff3cd41787b5d2 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:11.503736 8003 consensus_queue.cc:1035] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:11.506188 6922 raft_consensus.cc:1261] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:11.510916 6659 raft_consensus.cc:1261] T db14294a5205460dbdd0f7588d92c1b5 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:11.511307 7961 consensus_queue.cc:1035] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:11.512362 6791 raft_consensus.cc:1261] T db14294a5205460dbdd0f7588d92c1b5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:11.512496 7936 consensus_queue.cc:1035] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:11.513892 7947 consensus_queue.cc:1035] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 113.232 ms
time per row: 0.0566161 ms
I20240503 13:59:11.572166 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:11.577239 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 76e5e28dd13f475695def994b23c4695 (DEFAULT_TABLE table=rebalance_test_table_8 [id=2b50aeb14c1c4234955ff17e9bdb164f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:11.577340 6509 tablet_service.cc:1467] Processing CreateTablet for tablet 76e5e28dd13f475695def994b23c4695 (DEFAULT_TABLE table=rebalance_test_table_8 [id=2b50aeb14c1c4234955ff17e9bdb164f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:11.577610 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76e5e28dd13f475695def994b23c4695. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:11.577610 6509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76e5e28dd13f475695def994b23c4695. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:11.578258 6638 tablet_service.cc:1467] Processing CreateTablet for tablet 3a51d0527ff841b5b456962d591dcfe7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=2b50aeb14c1c4234955ff17e9bdb164f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:11.578327 6771 tablet_service.cc:1467] Processing CreateTablet for tablet 3a51d0527ff841b5b456962d591dcfe7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=2b50aeb14c1c4234955ff17e9bdb164f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:11.578485 6900 tablet_service.cc:1467] Processing CreateTablet for tablet 76e5e28dd13f475695def994b23c4695 (DEFAULT_TABLE table=rebalance_test_table_8 [id=2b50aeb14c1c4234955ff17e9bdb164f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:11.578606 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a51d0527ff841b5b456962d591dcfe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:11.578682 6901 tablet_service.cc:1467] Processing CreateTablet for tablet 5dcc6d09a5a8489a87f1b6bdce05a7ed (DEFAULT_TABLE table=rebalance_test_table_8 [id=2b50aeb14c1c4234955ff17e9bdb164f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:11.578759 6900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76e5e28dd13f475695def994b23c4695. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:11.578938 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dcc6d09a5a8489a87f1b6bdce05a7ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:11.579640 6507 tablet_service.cc:1467] Processing CreateTablet for tablet 3a51d0527ff841b5b456962d591dcfe7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=2b50aeb14c1c4234955ff17e9bdb164f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:11.580791 6507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a51d0527ff841b5b456962d591dcfe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:11.580825 7964 tablet_bootstrap.cc:492] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:11.581058 6770 tablet_service.cc:1467] Processing CreateTablet for tablet 5dcc6d09a5a8489a87f1b6bdce05a7ed (DEFAULT_TABLE table=rebalance_test_table_8 [id=2b50aeb14c1c4234955ff17e9bdb164f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:11.581059 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 5dcc6d09a5a8489a87f1b6bdce05a7ed (DEFAULT_TABLE table=rebalance_test_table_8 [id=2b50aeb14c1c4234955ff17e9bdb164f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:11.581247 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dcc6d09a5a8489a87f1b6bdce05a7ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:11.581251 6770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dcc6d09a5a8489a87f1b6bdce05a7ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:11.581728 7964 tablet_bootstrap.cc:654] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:11.582762 7963 tablet_bootstrap.cc:492] T 3a51d0527ff841b5b456962d591dcfe7 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:11.584132 7963 tablet_bootstrap.cc:654] T 3a51d0527ff841b5b456962d591dcfe7 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:11.585325 7963 tablet_bootstrap.cc:492] T 3a51d0527ff841b5b456962d591dcfe7 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:11.585340 7964 tablet_bootstrap.cc:492] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:11.585422 7963 ts_tablet_manager.cc:1367] T 3a51d0527ff841b5b456962d591dcfe7 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:59:11.585431 7964 ts_tablet_manager.cc:1367] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 13:59:11.585770 6638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a51d0527ff841b5b456962d591dcfe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:11.585870 7963 raft_consensus.cc:348] T 3a51d0527ff841b5b456962d591dcfe7 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:11.585978 7963 raft_consensus.cc:374] T 3a51d0527ff841b5b456962d591dcfe7 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:11.585932 7964 raft_consensus.cc:348] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:11.586031 7964 raft_consensus.cc:374] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:11.586027 7963 raft_consensus.cc:729] T 3a51d0527ff841b5b456962d591dcfe7 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:11.586072 7964 raft_consensus.cc:729] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:11.586153 7963 consensus_queue.cc:260] T 3a51d0527ff841b5b456962d591dcfe7 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:11.586203 7964 consensus_queue.cc:260] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:11.586405 7963 ts_tablet_manager.cc:1398] T 3a51d0527ff841b5b456962d591dcfe7 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:11.586573 7963 tablet_bootstrap.cc:492] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:11.586484 7964 ts_tablet_manager.cc:1398] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:11.587625 7964 tablet_bootstrap.cc:492] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:11.588325 7964 tablet_bootstrap.cc:654] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:11.588611 8024 tablet_bootstrap.cc:492] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:11.588987 8021 tablet_bootstrap.cc:492] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:11.589749 7964 tablet_bootstrap.cc:492] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:11.589838 7964 ts_tablet_manager.cc:1367] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:11.590232 7964 raft_consensus.cc:348] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:11.590392 7964 raft_consensus.cc:374] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:11.590431 7964 raft_consensus.cc:729] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:11.590441 8021 tablet_bootstrap.cc:654] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:11.590561 8024 tablet_bootstrap.cc:654] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:11.590567 7964 consensus_queue.cc:260] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:11.590866 7964 ts_tablet_manager.cc:1398] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:11.591493 7963 tablet_bootstrap.cc:654] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:11.592304 8024 tablet_bootstrap.cc:492] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:11.592341 8021 tablet_bootstrap.cc:492] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:11.592422 8021 ts_tablet_manager.cc:1367] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:11.592453 8024 ts_tablet_manager.cc:1367] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:11.592641 7963 tablet_bootstrap.cc:492] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:11.592734 7963 ts_tablet_manager.cc:1367] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 13:59:11.592898 8021 raft_consensus.cc:348] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:11.593010 8021 raft_consensus.cc:374] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:11.593053 8021 raft_consensus.cc:729] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:11.593171 8021 consensus_queue.cc:260] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:11.593201 7963 raft_consensus.cc:348] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:11.593295 7963 raft_consensus.cc:374] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:11.593339 7963 raft_consensus.cc:729] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:11.593453 8021 ts_tablet_manager.cc:1398] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:11.593474 7963 consensus_queue.cc:260] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:11.593644 8021 tablet_bootstrap.cc:492] T 76e5e28dd13f475695def994b23c4695 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:11.593798 7963 ts_tablet_manager.cc:1398] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:11.593967 7963 tablet_bootstrap.cc:492] T 76e5e28dd13f475695def994b23c4695 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:11.593927 8024 raft_consensus.cc:348] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:11.594157 8024 raft_consensus.cc:374] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:11.594210 8024 raft_consensus.cc:729] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:11.594336 8024 consensus_queue.cc:260] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:11.594529 8021 tablet_bootstrap.cc:654] T 76e5e28dd13f475695def994b23c4695 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:11.594624 8024 ts_tablet_manager.cc:1398] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:11.594822 8024 tablet_bootstrap.cc:492] T 3a51d0527ff841b5b456962d591dcfe7 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:11.594826 7963 tablet_bootstrap.cc:654] T 76e5e28dd13f475695def994b23c4695 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:11.595944 8024 tablet_bootstrap.cc:654] T 3a51d0527ff841b5b456962d591dcfe7 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:11.596307 8021 tablet_bootstrap.cc:492] T 76e5e28dd13f475695def994b23c4695 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:11.596377 8021 ts_tablet_manager.cc:1367] T 76e5e28dd13f475695def994b23c4695 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:11.596726 8021 raft_consensus.cc:348] T 76e5e28dd13f475695def994b23c4695 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:11.596835 8021 raft_consensus.cc:374] T 76e5e28dd13f475695def994b23c4695 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:11.596879 8021 raft_consensus.cc:729] T 76e5e28dd13f475695def994b23c4695 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:11.596994 8021 consensus_queue.cc:260] T 76e5e28dd13f475695def994b23c4695 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:11.597259 8021 ts_tablet_manager.cc:1398] T 76e5e28dd13f475695def994b23c4695 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:11.598863 7963 tablet_bootstrap.cc:492] T 76e5e28dd13f475695def994b23c4695 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:11.598942 7963 ts_tablet_manager.cc:1367] T 76e5e28dd13f475695def994b23c4695 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 13:59:11.599577 7963 raft_consensus.cc:348] T 76e5e28dd13f475695def994b23c4695 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:11.599685 7963 raft_consensus.cc:374] T 76e5e28dd13f475695def994b23c4695 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:11.599663 8024 tablet_bootstrap.cc:492] T 3a51d0527ff841b5b456962d591dcfe7 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:11.599733 7963 raft_consensus.cc:729] T 76e5e28dd13f475695def994b23c4695 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:11.599836 8024 ts_tablet_manager.cc:1367] T 3a51d0527ff841b5b456962d591dcfe7 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:11.599851 7963 consensus_queue.cc:260] T 76e5e28dd13f475695def994b23c4695 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:11.600114 7963 ts_tablet_manager.cc:1398] T 76e5e28dd13f475695def994b23c4695 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:11.600265 8024 raft_consensus.cc:348] T 3a51d0527ff841b5b456962d591dcfe7 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:11.600375 8024 raft_consensus.cc:374] T 3a51d0527ff841b5b456962d591dcfe7 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:11.600414 8024 raft_consensus.cc:729] T 3a51d0527ff841b5b456962d591dcfe7 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:11.600520 8024 consensus_queue.cc:260] T 3a51d0527ff841b5b456962d591dcfe7 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:11.600790 8024 ts_tablet_manager.cc:1398] T 3a51d0527ff841b5b456962d591dcfe7 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:11.647984 7758 raft_consensus.cc:482] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:11.648102 7758 raft_consensus.cc:504] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:11.648473 7758 leader_election.cc:288] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:11.648840 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e5e28dd13f475695def994b23c4695" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:11.648977 6528 raft_consensus.cc:2454] T 76e5e28dd13f475695def994b23c4695 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:11.649221 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e5e28dd13f475695def994b23c4695" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:11.649346 6922 raft_consensus.cc:2454] T 76e5e28dd13f475695def994b23c4695 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:11.649593 6595 leader_election.cc:302] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:11.649777 7758 raft_consensus.cc:2781] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:11.649837 7758 raft_consensus.cc:482] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:11.649897 7758 raft_consensus.cc:3037] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:11.650689 7758 raft_consensus.cc:504] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:11.650991 7758 leader_election.cc:288] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:11.651346 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e5e28dd13f475695def994b23c4695" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:11.651371 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e5e28dd13f475695def994b23c4695" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:11.651463 6922 raft_consensus.cc:3037] T 76e5e28dd13f475695def994b23c4695 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:11.651453 6528 raft_consensus.cc:3037] T 76e5e28dd13f475695def994b23c4695 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:11.652216 6922 raft_consensus.cc:2454] T 76e5e28dd13f475695def994b23c4695 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:11.652323 6528 raft_consensus.cc:2454] T 76e5e28dd13f475695def994b23c4695 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:11.652518 6595 leader_election.cc:302] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:11.652688 7758 raft_consensus.cc:2781] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:11.652856 7758 raft_consensus.cc:686] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 13:59:11.653002 7758 consensus_queue.cc:237] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:11.654781 6383 catalog_manager.cc:5484] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:11.752779 7890 raft_consensus.cc:482] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:11.752918 7890 raft_consensus.cc:504] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:11.753335 7890 leader_election.cc:288] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:11.753845 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcc6d09a5a8489a87f1b6bdce05a7ed" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:11.753844 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcc6d09a5a8489a87f1b6bdce05a7ed" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:11.753996 6922 raft_consensus.cc:2454] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:11.753996 6791 raft_consensus.cc:2454] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:11.754307 6464 leader_election.cc:302] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:11.754484 7890 raft_consensus.cc:2781] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:11.754580 7890 raft_consensus.cc:482] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:11.754611 7890 raft_consensus.cc:3037] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:11.755534 7890 raft_consensus.cc:504] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:11.755856 7890 leader_election.cc:288] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:11.756167 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcc6d09a5a8489a87f1b6bdce05a7ed" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:11.756167 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcc6d09a5a8489a87f1b6bdce05a7ed" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:11.756286 6791 raft_consensus.cc:3037] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:11.756286 6922 raft_consensus.cc:3037] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:11.757220 6791 raft_consensus.cc:2454] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:11.757220 6922 raft_consensus.cc:2454] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:11.757544 6463 leader_election.cc:302] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:11.757727 7890 raft_consensus.cc:2781] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:11.757915 7890 raft_consensus.cc:686] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Becoming Leader. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Running, Role: LEADER
I20240503 13:59:11.758069 7890 consensus_queue.cc:237] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:11.759941 6383 catalog_manager.cc:5484] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:11.816470 8003 raft_consensus.cc:482] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:11.816598 8003 raft_consensus.cc:504] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:11.817023 8003 leader_election.cc:288] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:11.817416 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a51d0527ff841b5b456962d591dcfe7" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:11.817425 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a51d0527ff841b5b456962d591dcfe7" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:11.817556 6528 raft_consensus.cc:2454] T 3a51d0527ff841b5b456962d591dcfe7 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:11.817570 6659 raft_consensus.cc:2454] T 3a51d0527ff841b5b456962d591dcfe7 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:11.817812 6726 leader_election.cc:302] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:11.818004 8003 raft_consensus.cc:2781] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:11.818051 8003 raft_consensus.cc:482] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:11.818095 8003 raft_consensus.cc:3037] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:11.819123 8003 raft_consensus.cc:504] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:11.819479 8003 leader_election.cc:288] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:11.819805 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a51d0527ff841b5b456962d591dcfe7" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:11.819816 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a51d0527ff841b5b456962d591dcfe7" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:11.819926 6528 raft_consensus.cc:3037] T 3a51d0527ff841b5b456962d591dcfe7 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:11.819926 6659 raft_consensus.cc:3037] T 3a51d0527ff841b5b456962d591dcfe7 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:11.820875 6659 raft_consensus.cc:2454] T 3a51d0527ff841b5b456962d591dcfe7 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:11.820884 6528 raft_consensus.cc:2454] T 3a51d0527ff841b5b456962d591dcfe7 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:11.821175 6727 leader_election.cc:302] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 4a85e492c8154e9881a9cc2a105d7e64, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:11.821341 8003 raft_consensus.cc:2781] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:11.821446 8003 raft_consensus.cc:686] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:11.821607 8003 consensus_queue.cc:237] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:11.823457 6383 catalog_manager.cc:5484] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:11.980244 8037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:11.980485 8037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:11.983413 8037 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 13:59:12.078444 6922 raft_consensus.cc:1261] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:12.078444 6791 raft_consensus.cc:1261] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:12.079156 7890 consensus_queue.cc:1035] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:12.079274 8034 consensus_queue.cc:1035] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:12.081072 6921 raft_consensus.cc:1261] T 76e5e28dd13f475695def994b23c4695 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:12.081436 7758 consensus_queue.cc:1035] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:12.081529 6528 raft_consensus.cc:1261] T 76e5e28dd13f475695def994b23c4695 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:12.084595 7758 consensus_queue.cc:1035] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:12.096464 6659 raft_consensus.cc:1261] T 3a51d0527ff841b5b456962d591dcfe7 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:12.096879 8003 consensus_queue.cc:1035] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:12.098078 6527 raft_consensus.cc:1261] T 3a51d0527ff841b5b456962d591dcfe7 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:12.098531 8003 consensus_queue.cc:1035] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 121.093 ms
time per row: 0.0605464 ms
I20240503 13:59:12.146853 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:12.151983 6901 tablet_service.cc:1467] Processing CreateTablet for tablet 21dd1c7418e54d30b665f57ef1842f9e (DEFAULT_TABLE table=rebalance_test_table_9 [id=010fdec94fd24d33b4513ebe788694ac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:12.152056 6770 tablet_service.cc:1467] Processing CreateTablet for tablet 21dd1c7418e54d30b665f57ef1842f9e (DEFAULT_TABLE table=rebalance_test_table_9 [id=010fdec94fd24d33b4513ebe788694ac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:12.152278 6770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21dd1c7418e54d30b665f57ef1842f9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.152279 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21dd1c7418e54d30b665f57ef1842f9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.152830 6638 tablet_service.cc:1467] Processing CreateTablet for tablet 21dd1c7418e54d30b665f57ef1842f9e (DEFAULT_TABLE table=rebalance_test_table_9 [id=010fdec94fd24d33b4513ebe788694ac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:12.152844 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 95fb9ad43cf34a78b98965035cd086ec (DEFAULT_TABLE table=rebalance_test_table_9 [id=010fdec94fd24d33b4513ebe788694ac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:12.153141 6638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21dd1c7418e54d30b665f57ef1842f9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.153321 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95fb9ad43cf34a78b98965035cd086ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.154546 6639 tablet_service.cc:1467] Processing CreateTablet for tablet 7c46822ccc1e48469c8d7f31b0bb0b8a (DEFAULT_TABLE table=rebalance_test_table_9 [id=010fdec94fd24d33b4513ebe788694ac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:12.154800 6639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c46822ccc1e48469c8d7f31b0bb0b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.154989 6770 tablet_service.cc:1467] Processing CreateTablet for tablet 95fb9ad43cf34a78b98965035cd086ec (DEFAULT_TABLE table=rebalance_test_table_9 [id=010fdec94fd24d33b4513ebe788694ac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:12.155208 6770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95fb9ad43cf34a78b98965035cd086ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.155452 8095 tablet_bootstrap.cc:492] T 21dd1c7418e54d30b665f57ef1842f9e P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:12.155753 6900 tablet_service.cc:1467] Processing CreateTablet for tablet 7c46822ccc1e48469c8d7f31b0bb0b8a (DEFAULT_TABLE table=rebalance_test_table_9 [id=010fdec94fd24d33b4513ebe788694ac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:12.155995 6900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c46822ccc1e48469c8d7f31b0bb0b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.156409 8095 tablet_bootstrap.cc:654] T 21dd1c7418e54d30b665f57ef1842f9e P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:12.157296 8094 tablet_bootstrap.cc:492] T 21dd1c7418e54d30b665f57ef1842f9e P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:12.157377 6509 tablet_service.cc:1467] Processing CreateTablet for tablet 7c46822ccc1e48469c8d7f31b0bb0b8a (DEFAULT_TABLE table=rebalance_test_table_9 [id=010fdec94fd24d33b4513ebe788694ac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:12.157613 6509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c46822ccc1e48469c8d7f31b0bb0b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.158048 8094 tablet_bootstrap.cc:654] T 21dd1c7418e54d30b665f57ef1842f9e P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:12.158314 8096 tablet_bootstrap.cc:492] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:12.159245 8096 tablet_bootstrap.cc:654] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:12.159658 8099 tablet_bootstrap.cc:492] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:12.160512 8099 tablet_bootstrap.cc:654] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:12.160601 8096 tablet_bootstrap.cc:492] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:12.160598 6901 tablet_service.cc:1467] Processing CreateTablet for tablet 95fb9ad43cf34a78b98965035cd086ec (DEFAULT_TABLE table=rebalance_test_table_9 [id=010fdec94fd24d33b4513ebe788694ac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:12.160681 8096 ts_tablet_manager.cc:1367] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:12.160815 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95fb9ad43cf34a78b98965035cd086ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.161087 8096 raft_consensus.cc:348] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:12.161212 8096 raft_consensus.cc:374] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:12.161247 8096 raft_consensus.cc:729] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:12.161377 8096 consensus_queue.cc:260] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:12.161651 8099 tablet_bootstrap.cc:492] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:12.161739 8099 ts_tablet_manager.cc:1367] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:12.162145 8099 raft_consensus.cc:348] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:12.162258 8099 raft_consensus.cc:374] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:12.162302 8099 raft_consensus.cc:729] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:12.162422 8099 consensus_queue.cc:260] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:12.162819 8099 ts_tablet_manager.cc:1398] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:12.163074 8094 tablet_bootstrap.cc:492] T 21dd1c7418e54d30b665f57ef1842f9e P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:12.163156 8094 ts_tablet_manager.cc:1367] T 21dd1c7418e54d30b665f57ef1842f9e P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 13:59:12.163450 8032 raft_consensus.cc:482] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:12.163576 8032 raft_consensus.cc:504] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:12.163673 8094 raft_consensus.cc:348] T 21dd1c7418e54d30b665f57ef1842f9e P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:12.163825 8094 raft_consensus.cc:374] T 21dd1c7418e54d30b665f57ef1842f9e P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:12.163859 8094 raft_consensus.cc:729] T 21dd1c7418e54d30b665f57ef1842f9e P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:12.163913 8032 leader_election.cc:288] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:12.164037 8094 consensus_queue.cc:260] T 21dd1c7418e54d30b665f57ef1842f9e P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:12.164259 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dd1c7418e54d30b665f57ef1842f9e" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:12.164364 6921 raft_consensus.cc:2454] T 21dd1c7418e54d30b665f57ef1842f9e P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:12.164361 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dd1c7418e54d30b665f57ef1842f9e" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:12.164539 8096 ts_tablet_manager.cc:1398] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:59:12.164650 6595 leader_election.cc:302] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:12.164904 8032 raft_consensus.cc:2781] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:12.164954 8032 raft_consensus.cc:482] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:12.164988 8032 raft_consensus.cc:3037] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
W20240503 13:59:12.165300 6594 leader_election.cc:341] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:59:12.165876 8032 raft_consensus.cc:504] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:12.166116 8094 ts_tablet_manager.cc:1398] T 21dd1c7418e54d30b665f57ef1842f9e P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:59:12.166227 8032 leader_election.cc:288] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:12.166501 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dd1c7418e54d30b665f57ef1842f9e" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:12.166599 8096 tablet_bootstrap.cc:492] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:12.166620 6921 raft_consensus.cc:3037] T 21dd1c7418e54d30b665f57ef1842f9e P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:12.166813 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dd1c7418e54d30b665f57ef1842f9e" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
W20240503 13:59:12.167147 6594 leader_election.cc:341] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:59:12.167469 6921 raft_consensus.cc:2454] T 21dd1c7418e54d30b665f57ef1842f9e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:12.167572 8096 tablet_bootstrap.cc:654] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:12.167814 8094 tablet_bootstrap.cc:492] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:12.167784 6595 leader_election.cc:302] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters: 4a85e492c8154e9881a9cc2a105d7e64
I20240503 13:59:12.167973 8032 raft_consensus.cc:2781] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:12.168047 8032 raft_consensus.cc:686] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 13:59:12.168167 8032 consensus_queue.cc:237] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:12.168603 8096 tablet_bootstrap.cc:492] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:12.168689 8096 ts_tablet_manager.cc:1367] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:12.168689 8094 tablet_bootstrap.cc:654] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:12.169071 8096 raft_consensus.cc:348] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:12.169162 8096 raft_consensus.cc:374] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:12.169191 8096 raft_consensus.cc:729] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:12.169296 8096 consensus_queue.cc:260] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:12.169528 8096 ts_tablet_manager.cc:1398] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:12.169701 8096 tablet_bootstrap.cc:492] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:12.169864 6384 catalog_manager.cc:5484] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:12.170518 8096 tablet_bootstrap.cc:654] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:12.171042 8094 tablet_bootstrap.cc:492] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:12.171137 8094 ts_tablet_manager.cc:1367] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:59:12.171586 8096 tablet_bootstrap.cc:492] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:12.171582 8094 raft_consensus.cc:348] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:12.171676 8094 raft_consensus.cc:374] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:12.171705 8094 raft_consensus.cc:729] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:12.171821 8094 consensus_queue.cc:260] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:12.171983 8096 ts_tablet_manager.cc:1367] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:12.172359 8096 raft_consensus.cc:348] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:12.172443 8096 raft_consensus.cc:374] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:12.172472 8096 raft_consensus.cc:729] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:12.172523 8094 ts_tablet_manager.cc:1398] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:12.172575 8096 consensus_queue.cc:260] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:12.172658 8094 tablet_bootstrap.cc:492] T 95fb9ad43cf34a78b98965035cd086ec P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:12.172865 8096 ts_tablet_manager.cc:1398] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:12.173398 8094 tablet_bootstrap.cc:654] T 95fb9ad43cf34a78b98965035cd086ec P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:12.174495 8095 tablet_bootstrap.cc:492] T 21dd1c7418e54d30b665f57ef1842f9e P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:12.174584 8095 ts_tablet_manager.cc:1367] T 21dd1c7418e54d30b665f57ef1842f9e P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 13:59:12.175091 8095 raft_consensus.cc:348] T 21dd1c7418e54d30b665f57ef1842f9e P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:12.175182 8094 tablet_bootstrap.cc:492] T 95fb9ad43cf34a78b98965035cd086ec P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:12.175204 8095 raft_consensus.cc:374] T 21dd1c7418e54d30b665f57ef1842f9e P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:12.175248 8094 ts_tablet_manager.cc:1367] T 95fb9ad43cf34a78b98965035cd086ec P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 13:59:12.175235 8095 raft_consensus.cc:729] T 21dd1c7418e54d30b665f57ef1842f9e P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:12.175397 8095 consensus_queue.cc:260] T 21dd1c7418e54d30b665f57ef1842f9e P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:12.175674 8095 ts_tablet_manager.cc:1398] T 21dd1c7418e54d30b665f57ef1842f9e P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:12.175750 8094 raft_consensus.cc:348] T 95fb9ad43cf34a78b98965035cd086ec P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:12.175835 8095 tablet_bootstrap.cc:492] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:12.175835 8094 raft_consensus.cc:374] T 95fb9ad43cf34a78b98965035cd086ec P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:12.175868 8094 raft_consensus.cc:729] T 95fb9ad43cf34a78b98965035cd086ec P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:12.175990 8094 consensus_queue.cc:260] T 95fb9ad43cf34a78b98965035cd086ec P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:12.176486 8094 ts_tablet_manager.cc:1398] T 95fb9ad43cf34a78b98965035cd086ec P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:12.176729 8095 tablet_bootstrap.cc:654] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:12.178180 8095 tablet_bootstrap.cc:492] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:12.178270 8095 ts_tablet_manager.cc:1367] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:12.178694 8095 raft_consensus.cc:348] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:12.178822 8095 raft_consensus.cc:374] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:12.178856 8095 raft_consensus.cc:729] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:12.178979 8095 consensus_queue.cc:260] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:12.179358 8095 ts_tablet_manager.cc:1398] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:12.225159 8091 raft_consensus.cc:482] T 21dd1c7418e54d30b665f57ef1842f9e P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:12.225275 8091 raft_consensus.cc:504] T 21dd1c7418e54d30b665f57ef1842f9e P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:12.225679 8091 leader_election.cc:288] T 21dd1c7418e54d30b665f57ef1842f9e P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:12.226073 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dd1c7418e54d30b665f57ef1842f9e" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:12.226161 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dd1c7418e54d30b665f57ef1842f9e" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:12.226332 6921 raft_consensus.cc:2379] T 21dd1c7418e54d30b665f57ef1842f9e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a85e492c8154e9881a9cc2a105d7e64 in current term 1: Already voted for candidate 2145d15ab2d74c0287839266f1411a43 in this term.
I20240503 13:59:12.226615 6726 leader_election.cc:302] T 21dd1c7418e54d30b665f57ef1842f9e P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64; no voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460
I20240503 13:59:12.226773 8091 raft_consensus.cc:3037] T 21dd1c7418e54d30b665f57ef1842f9e P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:12.227802 8091 raft_consensus.cc:2726] T 21dd1c7418e54d30b665f57ef1842f9e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 13:59:12.293139 8091 raft_consensus.cc:482] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:12.293303 8091 raft_consensus.cc:504] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:12.293710 8091 leader_election.cc:288] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:12.294179 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fb9ad43cf34a78b98965035cd086ec" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:12.294179 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fb9ad43cf34a78b98965035cd086ec" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:12.294329 6659 raft_consensus.cc:2454] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:12.294337 6921 raft_consensus.cc:2454] T 95fb9ad43cf34a78b98965035cd086ec P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:12.294602 6726 leader_election.cc:302] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:12.294782 8091 raft_consensus.cc:2781] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:12.294838 8091 raft_consensus.cc:482] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:12.294884 8091 raft_consensus.cc:3037] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:12.295689 8091 raft_consensus.cc:504] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:12.296016 8091 leader_election.cc:288] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:12.296355 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fb9ad43cf34a78b98965035cd086ec" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:12.296382 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fb9ad43cf34a78b98965035cd086ec" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:12.296464 6659 raft_consensus.cc:3037] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:12.296504 6921 raft_consensus.cc:3037] T 95fb9ad43cf34a78b98965035cd086ec P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:12.297245 6921 raft_consensus.cc:2454] T 95fb9ad43cf34a78b98965035cd086ec P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:12.297262 6659 raft_consensus.cc:2454] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:12.297477 6726 leader_election.cc:302] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:12.297672 8091 raft_consensus.cc:2781] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:12.297758 8091 raft_consensus.cc:686] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:12.297904 8091 consensus_queue.cc:237] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:12.299651 6384 catalog_manager.cc:5484] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:12.561209 7758 consensus_queue.cc:1035] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:12.564896 7758 consensus_queue.cc:1035] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:12.584705 7961 raft_consensus.cc:482] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:12.584821 7961 raft_consensus.cc:504] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:12.585222 7961 leader_election.cc:288] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:12.585566 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c46822ccc1e48469c8d7f31b0bb0b8a" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:12.585619 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c46822ccc1e48469c8d7f31b0bb0b8a" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:12.585709 6528 raft_consensus.cc:2454] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:12.585748 6659 raft_consensus.cc:2454] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:12.585997 6857 leader_election.cc:302] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:12.586189 7961 raft_consensus.cc:2781] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:12.586238 7961 raft_consensus.cc:482] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:12.586264 7961 raft_consensus.cc:3037] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:12.587142 7961 raft_consensus.cc:504] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:12.587491 7961 leader_election.cc:288] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:12.587898 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c46822ccc1e48469c8d7f31b0bb0b8a" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:12.587958 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c46822ccc1e48469c8d7f31b0bb0b8a" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:12.588038 6527 raft_consensus.cc:3037] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:12.588092 6659 raft_consensus.cc:3037] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:12.589020 6659 raft_consensus.cc:2454] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:12.589021 6527 raft_consensus.cc:2454] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:12.589334 6857 leader_election.cc:302] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:12.589533 7961 raft_consensus.cc:2781] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:12.589615 7961 raft_consensus.cc:686] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:12.589743 7961 consensus_queue.cc:237] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:12.591651 6384 catalog_manager.cc:5484] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:12.717536 8113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:12.717774 8113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:12.720736 8113 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240503 13:59:12.813639 6527 raft_consensus.cc:1261] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:12.813970 6659 raft_consensus.cc:1261] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:12.814389 7535 consensus_queue.cc:1035] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:12.814687 7535 consensus_queue.cc:1035] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:12.825623 8091 consensus_queue.cc:1035] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:12.836215 6660 raft_consensus.cc:1261] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:12.836786 8091 consensus_queue.cc:1035] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 126.276 ms
time per row: 0.063138 ms
I20240503 13:59:12.888234 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:12.893378 6770 tablet_service.cc:1467] Processing CreateTablet for tablet 3699694bf8dc46439ccf9b0aa1468369 (DEFAULT_TABLE table=rebalance_test_table_10 [id=bc0e9003ae9c41d9947f04fddb7235b3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:12.893424 6509 tablet_service.cc:1467] Processing CreateTablet for tablet 3699694bf8dc46439ccf9b0aa1468369 (DEFAULT_TABLE table=rebalance_test_table_10 [id=bc0e9003ae9c41d9947f04fddb7235b3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:12.893699 6509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3699694bf8dc46439ccf9b0aa1468369. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.893703 6770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3699694bf8dc46439ccf9b0aa1468369. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.894272 6771 tablet_service.cc:1467] Processing CreateTablet for tablet d014ee0844784387ade205aca2b676ca (DEFAULT_TABLE table=rebalance_test_table_10 [id=bc0e9003ae9c41d9947f04fddb7235b3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:12.894484 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d014ee0844784387ade205aca2b676ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.894531 6639 tablet_service.cc:1467] Processing CreateTablet for tablet 3699694bf8dc46439ccf9b0aa1468369 (DEFAULT_TABLE table=rebalance_test_table_10 [id=bc0e9003ae9c41d9947f04fddb7235b3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:12.894802 6639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3699694bf8dc46439ccf9b0aa1468369. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.896103 8168 tablet_bootstrap.cc:492] T 3699694bf8dc46439ccf9b0aa1468369 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:12.896201 6769 tablet_service.cc:1467] Processing CreateTablet for tablet ade8477f7bd04858a350c7849e1c5af9 (DEFAULT_TABLE table=rebalance_test_table_10 [id=bc0e9003ae9c41d9947f04fddb7235b3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:12.896418 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade8477f7bd04858a350c7849e1c5af9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.896652 6640 tablet_service.cc:1467] Processing CreateTablet for tablet ade8477f7bd04858a350c7849e1c5af9 (DEFAULT_TABLE table=rebalance_test_table_10 [id=bc0e9003ae9c41d9947f04fddb7235b3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:12.896881 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade8477f7bd04858a350c7849e1c5af9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.896999 8168 tablet_bootstrap.cc:654] T 3699694bf8dc46439ccf9b0aa1468369 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:12.898885 6901 tablet_service.cc:1467] Processing CreateTablet for tablet d014ee0844784387ade205aca2b676ca (DEFAULT_TABLE table=rebalance_test_table_10 [id=bc0e9003ae9c41d9947f04fddb7235b3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:12.899155 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d014ee0844784387ade205aca2b676ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.899580 8168 tablet_bootstrap.cc:492] T 3699694bf8dc46439ccf9b0aa1468369 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:12.899696 8168 ts_tablet_manager.cc:1367] T 3699694bf8dc46439ccf9b0aa1468369 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:12.900408 8168 raft_consensus.cc:348] T 3699694bf8dc46439ccf9b0aa1468369 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:12.900589 8168 raft_consensus.cc:374] T 3699694bf8dc46439ccf9b0aa1468369 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:12.900656 8168 raft_consensus.cc:729] T 3699694bf8dc46439ccf9b0aa1468369 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:12.900820 8168 consensus_queue.cc:260] T 3699694bf8dc46439ccf9b0aa1468369 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:12.901275 8168 ts_tablet_manager.cc:1398] T 3699694bf8dc46439ccf9b0aa1468369 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:12.901278 6508 tablet_service.cc:1467] Processing CreateTablet for tablet ade8477f7bd04858a350c7849e1c5af9 (DEFAULT_TABLE table=rebalance_test_table_10 [id=bc0e9003ae9c41d9947f04fddb7235b3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:12.901466 8168 tablet_bootstrap.cc:492] T d014ee0844784387ade205aca2b676ca P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:12.901541 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade8477f7bd04858a350c7849e1c5af9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.901598 6509 tablet_service.cc:1467] Processing CreateTablet for tablet d014ee0844784387ade205aca2b676ca (DEFAULT_TABLE table=rebalance_test_table_10 [id=bc0e9003ae9c41d9947f04fddb7235b3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:12.901835 6509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d014ee0844784387ade205aca2b676ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.902165 8172 tablet_bootstrap.cc:492] T d014ee0844784387ade205aca2b676ca P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:12.902498 8168 tablet_bootstrap.cc:654] T d014ee0844784387ade205aca2b676ca P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:12.903053 8172 tablet_bootstrap.cc:654] T d014ee0844784387ade205aca2b676ca P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:12.903313 8171 tablet_bootstrap.cc:492] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:12.903445 8169 tablet_bootstrap.cc:492] T 3699694bf8dc46439ccf9b0aa1468369 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:12.904841 8169 tablet_bootstrap.cc:654] T 3699694bf8dc46439ccf9b0aa1468369 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:12.905373 8171 tablet_bootstrap.cc:654] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:12.905382 8168 tablet_bootstrap.cc:492] T d014ee0844784387ade205aca2b676ca P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:12.905449 8168 ts_tablet_manager.cc:1367] T d014ee0844784387ade205aca2b676ca P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:12.905817 8169 tablet_bootstrap.cc:492] T 3699694bf8dc46439ccf9b0aa1468369 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:12.905910 8169 ts_tablet_manager.cc:1367] T 3699694bf8dc46439ccf9b0aa1468369 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:12.905874 8168 raft_consensus.cc:348] T d014ee0844784387ade205aca2b676ca P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:12.905998 8168 raft_consensus.cc:374] T d014ee0844784387ade205aca2b676ca P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:12.906049 8168 raft_consensus.cc:729] T d014ee0844784387ade205aca2b676ca P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:12.906203 8168 consensus_queue.cc:260] T d014ee0844784387ade205aca2b676ca P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:12.906352 8169 raft_consensus.cc:348] T 3699694bf8dc46439ccf9b0aa1468369 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:12.906464 8169 raft_consensus.cc:374] T 3699694bf8dc46439ccf9b0aa1468369 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:12.906507 8169 raft_consensus.cc:729] T 3699694bf8dc46439ccf9b0aa1468369 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:12.906600 8168 ts_tablet_manager.cc:1398] T d014ee0844784387ade205aca2b676ca P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:12.906625 8169 consensus_queue.cc:260] T 3699694bf8dc46439ccf9b0aa1468369 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:12.906780 8168 tablet_bootstrap.cc:492] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:12.906917 8169 ts_tablet_manager.cc:1398] T 3699694bf8dc46439ccf9b0aa1468369 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:12.907079 8169 tablet_bootstrap.cc:492] T ade8477f7bd04858a350c7849e1c5af9 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:12.907588 8168 tablet_bootstrap.cc:654] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:12.907786 8172 tablet_bootstrap.cc:492] T d014ee0844784387ade205aca2b676ca P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:12.907903 8172 ts_tablet_manager.cc:1367] T d014ee0844784387ade205aca2b676ca P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:59:12.908442 8172 raft_consensus.cc:348] T d014ee0844784387ade205aca2b676ca P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:12.908548 8172 raft_consensus.cc:374] T d014ee0844784387ade205aca2b676ca P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:12.908582 8172 raft_consensus.cc:729] T d014ee0844784387ade205aca2b676ca P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:12.908696 8172 consensus_queue.cc:260] T d014ee0844784387ade205aca2b676ca P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:12.908990 8172 ts_tablet_manager.cc:1398] T d014ee0844784387ade205aca2b676ca P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:12.909366 8168 tablet_bootstrap.cc:492] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:12.909440 8168 ts_tablet_manager.cc:1367] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:12.909821 8168 raft_consensus.cc:348] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:12.909901 8168 raft_consensus.cc:374] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:12.909930 8168 raft_consensus.cc:729] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:12.910050 8168 consensus_queue.cc:260] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:12.910306 8168 ts_tablet_manager.cc:1398] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:12.910506 8169 tablet_bootstrap.cc:654] T ade8477f7bd04858a350c7849e1c5af9 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:12.911597 8169 tablet_bootstrap.cc:492] T ade8477f7bd04858a350c7849e1c5af9 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:12.911674 8169 ts_tablet_manager.cc:1367] T ade8477f7bd04858a350c7849e1c5af9 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:59:12.911799 8171 tablet_bootstrap.cc:492] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:12.911885 8171 ts_tablet_manager.cc:1367] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:59:12.912094 8169 raft_consensus.cc:348] T ade8477f7bd04858a350c7849e1c5af9 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:12.912199 8169 raft_consensus.cc:374] T ade8477f7bd04858a350c7849e1c5af9 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:12.912240 8169 raft_consensus.cc:729] T ade8477f7bd04858a350c7849e1c5af9 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:12.912359 8169 consensus_queue.cc:260] T ade8477f7bd04858a350c7849e1c5af9 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:12.912341 8171 raft_consensus.cc:348] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:12.912446 8171 raft_consensus.cc:374] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:12.912475 8171 raft_consensus.cc:729] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:12.912612 8169 ts_tablet_manager.cc:1398] T ade8477f7bd04858a350c7849e1c5af9 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:12.912604 8171 consensus_queue.cc:260] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:12.912737 8169 tablet_bootstrap.cc:492] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:12.912909 8171 ts_tablet_manager.cc:1398] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:12.913084 8171 tablet_bootstrap.cc:492] T ade8477f7bd04858a350c7849e1c5af9 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:12.913580 8169 tablet_bootstrap.cc:654] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:12.914022 8171 tablet_bootstrap.cc:654] T ade8477f7bd04858a350c7849e1c5af9 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:12.914817 8169 tablet_bootstrap.cc:492] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:12.915695 8171 tablet_bootstrap.cc:492] T ade8477f7bd04858a350c7849e1c5af9 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:12.915781 8171 ts_tablet_manager.cc:1367] T ade8477f7bd04858a350c7849e1c5af9 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:12.915992 8169 ts_tablet_manager.cc:1367] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:12.916241 8171 raft_consensus.cc:348] T ade8477f7bd04858a350c7849e1c5af9 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:12.916379 8171 raft_consensus.cc:374] T ade8477f7bd04858a350c7849e1c5af9 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:12.916343 8169 raft_consensus.cc:348] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:12.916431 8171 raft_consensus.cc:729] T ade8477f7bd04858a350c7849e1c5af9 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:12.916435 8169 raft_consensus.cc:374] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:12.916476 8169 raft_consensus.cc:729] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:12.916574 8171 consensus_queue.cc:260] T ade8477f7bd04858a350c7849e1c5af9 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:12.916638 8169 consensus_queue.cc:260] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:12.916857 8171 ts_tablet_manager.cc:1398] T ade8477f7bd04858a350c7849e1c5af9 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:12.916949 8169 ts_tablet_manager.cc:1398] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:13.023626 8091 raft_consensus.cc:482] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:13.023767 8091 raft_consensus.cc:504] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:13.024219 8091 leader_election.cc:288] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:13.024559 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade8477f7bd04858a350c7849e1c5af9" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:13.024708 6659 raft_consensus.cc:2454] T ade8477f7bd04858a350c7849e1c5af9 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:13.024722 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade8477f7bd04858a350c7849e1c5af9" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:13.024838 6527 raft_consensus.cc:2454] T ade8477f7bd04858a350c7849e1c5af9 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:13.025048 6727 leader_election.cc:302] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:13.025233 8091 raft_consensus.cc:2781] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:13.025300 8091 raft_consensus.cc:482] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:13.025350 8091 raft_consensus.cc:3037] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:13.026228 8091 raft_consensus.cc:504] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:13.026532 8091 leader_election.cc:288] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:13.026835 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade8477f7bd04858a350c7849e1c5af9" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:13.026858 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade8477f7bd04858a350c7849e1c5af9" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:13.026937 6659 raft_consensus.cc:3037] T ade8477f7bd04858a350c7849e1c5af9 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:13.026975 6527 raft_consensus.cc:3037] T ade8477f7bd04858a350c7849e1c5af9 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:13.027729 6659 raft_consensus.cc:2454] T ade8477f7bd04858a350c7849e1c5af9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:13.027746 6527 raft_consensus.cc:2454] T ade8477f7bd04858a350c7849e1c5af9 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:13.027993 6727 leader_election.cc:302] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:13.028167 8091 raft_consensus.cc:2781] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:13.028249 8091 raft_consensus.cc:686] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:13.028352 8091 consensus_queue.cc:237] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:13.030082 6378 catalog_manager.cc:5484] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:13.195631 8110 raft_consensus.cc:482] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:13.195797 8110 raft_consensus.cc:504] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:13.196167 8110 leader_election.cc:288] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:13.196452 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3699694bf8dc46439ccf9b0aa1468369" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:13.196491 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3699694bf8dc46439ccf9b0aa1468369" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:13.196584 6791 raft_consensus.cc:2454] T 3699694bf8dc46439ccf9b0aa1468369 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:13.196597 6528 raft_consensus.cc:2454] T 3699694bf8dc46439ccf9b0aa1468369 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:13.196849 8034 raft_consensus.cc:482] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:13.196941 8034 raft_consensus.cc:504] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:13.197299 8034 leader_election.cc:288] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:13.197623 6595 leader_election.cc:302] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:13.197630 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d014ee0844784387ade205aca2b676ca" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:13.197722 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d014ee0844784387ade205aca2b676ca" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:13.197780 8110 raft_consensus.cc:2781] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:13.197775 6922 raft_consensus.cc:2454] T d014ee0844784387ade205aca2b676ca P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:13.197830 8110 raft_consensus.cc:482] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:13.197865 8110 raft_consensus.cc:3037] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:13.197841 6791 raft_consensus.cc:2454] T d014ee0844784387ade205aca2b676ca P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:13.198201 6464 leader_election.cc:302] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:13.198447 8034 raft_consensus.cc:2781] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:13.198510 8034 raft_consensus.cc:482] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:13.198565 8034 raft_consensus.cc:3037] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:13.198890 8110 raft_consensus.cc:504] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:13.199191 8110 leader_election.cc:288] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:13.199517 8034 raft_consensus.cc:504] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:13.199813 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3699694bf8dc46439ccf9b0aa1468369" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:13.199906 8034 leader_election.cc:288] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:13.200004 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3699694bf8dc46439ccf9b0aa1468369" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:13.200120 6791 raft_consensus.cc:3037] T 3699694bf8dc46439ccf9b0aa1468369 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:13.200290 6789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d014ee0844784387ade205aca2b676ca" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:13.200428 6789 raft_consensus.cc:3037] T d014ee0844784387ade205aca2b676ca P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:13.200408 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d014ee0844784387ade205aca2b676ca" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:13.200503 6922 raft_consensus.cc:3037] T d014ee0844784387ade205aca2b676ca P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:13.201076 6791 raft_consensus.cc:2454] T 3699694bf8dc46439ccf9b0aa1468369 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:13.201177 6789 raft_consensus.cc:2454] T d014ee0844784387ade205aca2b676ca P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:13.201457 6594 leader_election.cc:302] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:13.201467 6463 leader_election.cc:302] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:13.201483 6922 raft_consensus.cc:2454] T d014ee0844784387ade205aca2b676ca P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:13.201632 8110 raft_consensus.cc:2781] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:13.199921 6528 raft_consensus.cc:3037] T 3699694bf8dc46439ccf9b0aa1468369 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:13.201718 8110 raft_consensus.cc:686] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 13:59:13.201763 8182 raft_consensus.cc:2781] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:13.201835 8182 raft_consensus.cc:686] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Becoming Leader. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Running, Role: LEADER
I20240503 13:59:13.201874 8110 consensus_queue.cc:237] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:13.201983 8182 consensus_queue.cc:237] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:13.202558 6528 raft_consensus.cc:2454] T 3699694bf8dc46439ccf9b0aa1468369 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:13.203902 6383 catalog_manager.cc:5484] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:13.203946 6378 catalog_manager.cc:5484] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } }
W20240503 13:59:13.378183 8186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:13.378446 8186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:13.381542 8186 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240503 13:59:13.416623 8091 consensus_queue.cc:1035] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:13.424109 8160 consensus_queue.cc:1035] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:13.480644 6528 raft_consensus.cc:1261] T 3699694bf8dc46439ccf9b0aa1468369 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:13.481003 6789 raft_consensus.cc:1261] T 3699694bf8dc46439ccf9b0aa1468369 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:13.481057 8032 consensus_queue.cc:1035] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:13.481384 8032 consensus_queue.cc:1035] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:13.482141 6922 raft_consensus.cc:1261] T d014ee0844784387ade205aca2b676ca P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:13.482654 8182 consensus_queue.cc:1035] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:13.482820 6789 raft_consensus.cc:1261] T d014ee0844784387ade205aca2b676ca P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:13.484508 8182 consensus_queue.cc:1035] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 122.813 ms
time per row: 0.0614067 ms
I20240503 13:59:13.548519 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:13.553730 6901 tablet_service.cc:1467] Processing CreateTablet for tablet cf7323c2594e4cd5967c3d38e4120815 (DEFAULT_TABLE table=rebalance_test_table_11 [id=3e15c9008ce84692a697ae9651a18574]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:13.554059 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7323c2594e4cd5967c3d38e4120815. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:13.554531 6769 tablet_service.cc:1467] Processing CreateTablet for tablet 342229593b184019ab915c0389b184b4 (DEFAULT_TABLE table=rebalance_test_table_11 [id=3e15c9008ce84692a697ae9651a18574]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:13.554601 6771 tablet_service.cc:1467] Processing CreateTablet for tablet 3856ff74b3bf4f608e3c6cdd06ce144e (DEFAULT_TABLE table=rebalance_test_table_11 [id=3e15c9008ce84692a697ae9651a18574]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:13.554786 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 342229593b184019ab915c0389b184b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:13.554841 6900 tablet_service.cc:1467] Processing CreateTablet for tablet 342229593b184019ab915c0389b184b4 (DEFAULT_TABLE table=rebalance_test_table_11 [id=3e15c9008ce84692a697ae9651a18574]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:13.554924 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3856ff74b3bf4f608e3c6cdd06ce144e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:13.555055 6900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 342229593b184019ab915c0389b184b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:13.557015 6902 tablet_service.cc:1467] Processing CreateTablet for tablet 3856ff74b3bf4f608e3c6cdd06ce144e (DEFAULT_TABLE table=rebalance_test_table_11 [id=3e15c9008ce84692a697ae9651a18574]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:13.557044 8246 tablet_bootstrap.cc:492] T 342229593b184019ab915c0389b184b4 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:13.557233 6902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3856ff74b3bf4f608e3c6cdd06ce144e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:13.557514 6640 tablet_service.cc:1467] Processing CreateTablet for tablet cf7323c2594e4cd5967c3d38e4120815 (DEFAULT_TABLE table=rebalance_test_table_11 [id=3e15c9008ce84692a697ae9651a18574]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:13.557564 6639 tablet_service.cc:1467] Processing CreateTablet for tablet 3856ff74b3bf4f608e3c6cdd06ce144e (DEFAULT_TABLE table=rebalance_test_table_11 [id=3e15c9008ce84692a697ae9651a18574]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:13.557643 8245 tablet_bootstrap.cc:492] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:13.557744 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7323c2594e4cd5967c3d38e4120815. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:13.557868 6639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3856ff74b3bf4f608e3c6cdd06ce144e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:13.558466 8245 tablet_bootstrap.cc:654] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:13.553807 6509 tablet_service.cc:1467] Processing CreateTablet for tablet cf7323c2594e4cd5967c3d38e4120815 (DEFAULT_TABLE table=rebalance_test_table_11 [id=3e15c9008ce84692a697ae9651a18574]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:13.559365 6509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7323c2594e4cd5967c3d38e4120815. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:13.560276 8248 tablet_bootstrap.cc:492] T 3856ff74b3bf4f608e3c6cdd06ce144e P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:13.560520 8246 tablet_bootstrap.cc:654] T 342229593b184019ab915c0389b184b4 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:13.561168 8248 tablet_bootstrap.cc:654] T 3856ff74b3bf4f608e3c6cdd06ce144e P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:13.556712 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 342229593b184019ab915c0389b184b4 (DEFAULT_TABLE table=rebalance_test_table_11 [id=3e15c9008ce84692a697ae9651a18574]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:13.561585 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 342229593b184019ab915c0389b184b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:13.561833 8246 tablet_bootstrap.cc:492] T 342229593b184019ab915c0389b184b4 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:13.561877 8250 tablet_bootstrap.cc:492] T cf7323c2594e4cd5967c3d38e4120815 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:13.561997 8246 ts_tablet_manager.cc:1367] T 342229593b184019ab915c0389b184b4 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:13.562590 8246 raft_consensus.cc:348] T 342229593b184019ab915c0389b184b4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:13.562808 8246 raft_consensus.cc:374] T 342229593b184019ab915c0389b184b4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:13.562888 8246 raft_consensus.cc:729] T 342229593b184019ab915c0389b184b4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:13.563081 8246 consensus_queue.cc:260] T 342229593b184019ab915c0389b184b4 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:13.563508 8246 ts_tablet_manager.cc:1398] T 342229593b184019ab915c0389b184b4 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:13.563763 8246 tablet_bootstrap.cc:492] T 3856ff74b3bf4f608e3c6cdd06ce144e P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:13.564571 8245 tablet_bootstrap.cc:492] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:13.564636 8246 tablet_bootstrap.cc:654] T 3856ff74b3bf4f608e3c6cdd06ce144e P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:13.564671 8245 ts_tablet_manager.cc:1367] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:59:13.565165 8245 raft_consensus.cc:348] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:13.565292 8245 raft_consensus.cc:374] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:13.565332 8245 raft_consensus.cc:729] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:13.565459 8245 consensus_queue.cc:260] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:13.565624 8246 tablet_bootstrap.cc:492] T 3856ff74b3bf4f608e3c6cdd06ce144e P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:13.565685 8246 ts_tablet_manager.cc:1367] T 3856ff74b3bf4f608e3c6cdd06ce144e P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:13.565742 8245 ts_tablet_manager.cc:1398] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:13.565912 8245 tablet_bootstrap.cc:492] T 342229593b184019ab915c0389b184b4 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:13.566061 8246 raft_consensus.cc:348] T 3856ff74b3bf4f608e3c6cdd06ce144e P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:13.566138 8246 raft_consensus.cc:374] T 3856ff74b3bf4f608e3c6cdd06ce144e P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:13.566184 8246 raft_consensus.cc:729] T 3856ff74b3bf4f608e3c6cdd06ce144e P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:13.566304 8246 consensus_queue.cc:260] T 3856ff74b3bf4f608e3c6cdd06ce144e P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:13.566552 8246 ts_tablet_manager.cc:1398] T 3856ff74b3bf4f608e3c6cdd06ce144e P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:13.566715 8245 tablet_bootstrap.cc:654] T 342229593b184019ab915c0389b184b4 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:13.568567 8248 tablet_bootstrap.cc:492] T 3856ff74b3bf4f608e3c6cdd06ce144e P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:13.568655 8248 ts_tablet_manager.cc:1367] T 3856ff74b3bf4f608e3c6cdd06ce144e P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:59:13.569118 8248 raft_consensus.cc:348] T 3856ff74b3bf4f608e3c6cdd06ce144e P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:13.569219 8248 raft_consensus.cc:374] T 3856ff74b3bf4f608e3c6cdd06ce144e P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:13.569249 8248 raft_consensus.cc:729] T 3856ff74b3bf4f608e3c6cdd06ce144e P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:13.569362 8248 consensus_queue.cc:260] T 3856ff74b3bf4f608e3c6cdd06ce144e P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:13.569664 8248 ts_tablet_manager.cc:1398] T 3856ff74b3bf4f608e3c6cdd06ce144e P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:13.569772 8250 tablet_bootstrap.cc:654] T cf7323c2594e4cd5967c3d38e4120815 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:13.570180 8248 tablet_bootstrap.cc:492] T cf7323c2594e4cd5967c3d38e4120815 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:13.571012 8248 tablet_bootstrap.cc:654] T cf7323c2594e4cd5967c3d38e4120815 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:13.572053 8248 tablet_bootstrap.cc:492] T cf7323c2594e4cd5967c3d38e4120815 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:13.572136 8248 ts_tablet_manager.cc:1367] T cf7323c2594e4cd5967c3d38e4120815 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:13.572407 8245 tablet_bootstrap.cc:492] T 342229593b184019ab915c0389b184b4 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:13.572480 8245 ts_tablet_manager.cc:1367] T 342229593b184019ab915c0389b184b4 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 13:59:13.572537 8250 tablet_bootstrap.cc:492] T cf7323c2594e4cd5967c3d38e4120815 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:13.572542 8248 raft_consensus.cc:348] T cf7323c2594e4cd5967c3d38e4120815 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:13.572628 8250 ts_tablet_manager.cc:1367] T cf7323c2594e4cd5967c3d38e4120815 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 13:59:13.572633 8248 raft_consensus.cc:374] T cf7323c2594e4cd5967c3d38e4120815 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:13.572664 8248 raft_consensus.cc:729] T cf7323c2594e4cd5967c3d38e4120815 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:13.572779 8248 consensus_queue.cc:260] T cf7323c2594e4cd5967c3d38e4120815 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:13.572924 8245 raft_consensus.cc:348] T 342229593b184019ab915c0389b184b4 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:13.573009 8245 raft_consensus.cc:374] T 342229593b184019ab915c0389b184b4 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:13.573036 8245 raft_consensus.cc:729] T 342229593b184019ab915c0389b184b4 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:13.573065 8248 ts_tablet_manager.cc:1398] T cf7323c2594e4cd5967c3d38e4120815 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:13.573159 8245 consensus_queue.cc:260] T 342229593b184019ab915c0389b184b4 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:13.573185 8250 raft_consensus.cc:348] T cf7323c2594e4cd5967c3d38e4120815 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:13.573293 8250 raft_consensus.cc:374] T cf7323c2594e4cd5967c3d38e4120815 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:13.573333 8250 raft_consensus.cc:729] T cf7323c2594e4cd5967c3d38e4120815 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:13.573413 8245 ts_tablet_manager.cc:1398] T 342229593b184019ab915c0389b184b4 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:13.573452 8250 consensus_queue.cc:260] T cf7323c2594e4cd5967c3d38e4120815 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:13.573546 8245 tablet_bootstrap.cc:492] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:13.573717 8250 ts_tablet_manager.cc:1398] T cf7323c2594e4cd5967c3d38e4120815 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:13.573853 8250 tablet_bootstrap.cc:492] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:13.574218 8245 tablet_bootstrap.cc:654] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:13.574654 8250 tablet_bootstrap.cc:654] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:13.575155 8245 tablet_bootstrap.cc:492] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:13.575229 8245 ts_tablet_manager.cc:1367] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:13.575848 8245 raft_consensus.cc:348] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:13.575918 8245 raft_consensus.cc:374] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:13.575937 8245 raft_consensus.cc:729] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:13.576043 8245 consensus_queue.cc:260] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:13.576401 8245 ts_tablet_manager.cc:1398] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:13.576822 8250 tablet_bootstrap.cc:492] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:13.576915 8250 ts_tablet_manager.cc:1367] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:13.577327 8250 raft_consensus.cc:348] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:13.577426 8250 raft_consensus.cc:374] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:13.577463 8250 raft_consensus.cc:729] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:13.577572 8250 consensus_queue.cc:260] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:13.577852 8250 ts_tablet_manager.cc:1398] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:13.585335 8253 raft_consensus.cc:482] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:13.585448 8253 raft_consensus.cc:504] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:13.585860 8253 leader_election.cc:288] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:13.586247 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3856ff74b3bf4f608e3c6cdd06ce144e" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:13.586284 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3856ff74b3bf4f608e3c6cdd06ce144e" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:13.586395 6659 raft_consensus.cc:2454] T 3856ff74b3bf4f608e3c6cdd06ce144e P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:13.586417 6791 raft_consensus.cc:2454] T 3856ff74b3bf4f608e3c6cdd06ce144e P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:13.586782 6856 leader_election.cc:302] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:13.587013 8253 raft_consensus.cc:2781] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:13.587069 8253 raft_consensus.cc:482] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:13.587110 8253 raft_consensus.cc:3037] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:13.587967 8253 raft_consensus.cc:504] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:13.588290 8253 leader_election.cc:288] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:13.588613 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3856ff74b3bf4f608e3c6cdd06ce144e" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:13.588733 6791 raft_consensus.cc:3037] T 3856ff74b3bf4f608e3c6cdd06ce144e P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:13.588717 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3856ff74b3bf4f608e3c6cdd06ce144e" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:13.588833 6659 raft_consensus.cc:3037] T 3856ff74b3bf4f608e3c6cdd06ce144e P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:13.589475 6791 raft_consensus.cc:2454] T 3856ff74b3bf4f608e3c6cdd06ce144e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:13.589530 6659 raft_consensus.cc:2454] T 3856ff74b3bf4f608e3c6cdd06ce144e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:13.589776 6856 leader_election.cc:302] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:13.589973 7535 raft_consensus.cc:2781] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:13.590050 7535 raft_consensus.cc:686] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:13.590183 7535 consensus_queue.cc:237] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:13.592155 6383 catalog_manager.cc:5484] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:13.762593 8034 raft_consensus.cc:482] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:13.762708 8034 raft_consensus.cc:504] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:13.763103 8034 leader_election.cc:288] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:13.763487 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "342229593b184019ab915c0389b184b4" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:13.763489 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "342229593b184019ab915c0389b184b4" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:13.763633 6922 raft_consensus.cc:2454] T 342229593b184019ab915c0389b184b4 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:13.763639 6791 raft_consensus.cc:2454] T 342229593b184019ab915c0389b184b4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:13.763994 6464 leader_election.cc:302] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:13.764185 8034 raft_consensus.cc:2781] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:13.764267 8034 raft_consensus.cc:482] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:13.764338 8034 raft_consensus.cc:3037] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:13.765256 8034 raft_consensus.cc:504] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:13.765551 8034 leader_election.cc:288] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:13.765861 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "342229593b184019ab915c0389b184b4" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:13.765861 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "342229593b184019ab915c0389b184b4" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:13.765973 6922 raft_consensus.cc:3037] T 342229593b184019ab915c0389b184b4 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:13.765973 6791 raft_consensus.cc:3037] T 342229593b184019ab915c0389b184b4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:13.766923 6922 raft_consensus.cc:2454] T 342229593b184019ab915c0389b184b4 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:13.766923 6791 raft_consensus.cc:2454] T 342229593b184019ab915c0389b184b4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:13.767170 6464 leader_election.cc:302] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:13.767359 8034 raft_consensus.cc:2781] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:13.767472 8034 raft_consensus.cc:686] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Becoming Leader. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Running, Role: LEADER
I20240503 13:59:13.767606 8034 consensus_queue.cc:237] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:13.769480 6383 catalog_manager.cc:5484] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:13.811887 7535 raft_consensus.cc:482] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:13.812002 7535 raft_consensus.cc:504] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:13.812355 7535 leader_election.cc:288] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:13.812685 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7323c2594e4cd5967c3d38e4120815" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:13.812713 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7323c2594e4cd5967c3d38e4120815" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:13.812811 6659 raft_consensus.cc:2454] T cf7323c2594e4cd5967c3d38e4120815 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:13.812849 6528 raft_consensus.cc:2454] T cf7323c2594e4cd5967c3d38e4120815 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:13.813107 6858 leader_election.cc:302] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:13.813270 7535 raft_consensus.cc:2781] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:13.813318 7535 raft_consensus.cc:482] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:13.813382 7535 raft_consensus.cc:3037] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:13.816195 7535 raft_consensus.cc:504] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:13.816550 7535 leader_election.cc:288] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:13.816926 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7323c2594e4cd5967c3d38e4120815" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:13.816929 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7323c2594e4cd5967c3d38e4120815" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:13.817035 6528 raft_consensus.cc:3037] T cf7323c2594e4cd5967c3d38e4120815 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:13.817035 6659 raft_consensus.cc:3037] T cf7323c2594e4cd5967c3d38e4120815 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:13.817854 6659 raft_consensus.cc:2454] T cf7323c2594e4cd5967c3d38e4120815 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:13.818032 6528 raft_consensus.cc:2454] T cf7323c2594e4cd5967c3d38e4120815 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:13.818156 6858 leader_election.cc:302] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:13.818339 7535 raft_consensus.cc:2781] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:13.818523 7535 raft_consensus.cc:686] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:13.818670 7535 consensus_queue.cc:237] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:13.820521 6383 catalog_manager.cc:5484] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:13.968297 8262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:13.968566 8262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:13.971668 8262 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240503 13:59:13.992919 7535 consensus_queue.cc:1035] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:14.000025 8253 consensus_queue.cc:1035] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:14.069114 6922 raft_consensus.cc:1261] T 342229593b184019ab915c0389b184b4 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:14.069557 6791 raft_consensus.cc:1261] T 342229593b184019ab915c0389b184b4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:14.070490 8182 consensus_queue.cc:1035] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:14.070784 8182 consensus_queue.cc:1035] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:14.080675 6527 raft_consensus.cc:1261] T cf7323c2594e4cd5967c3d38e4120815 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:14.080969 6659 raft_consensus.cc:1261] T cf7323c2594e4cd5967c3d38e4120815 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:14.081422 8253 consensus_queue.cc:1035] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:14.081722 8253 consensus_queue.cc:1035] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 113.976 ms
time per row: 0.056988 ms
I20240503 13:59:14.136814 6379 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:14.142488 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 629248af74c2468ba52fd6104e2d98d9 (DEFAULT_TABLE table=rebalance_test_table_12 [id=e0c6bd9b27794a2887eccea514c01058]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:14.142693 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 629248af74c2468ba52fd6104e2d98d9 (DEFAULT_TABLE table=rebalance_test_table_12 [id=e0c6bd9b27794a2887eccea514c01058]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:14.142866 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 629248af74c2468ba52fd6104e2d98d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:14.143003 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 629248af74c2468ba52fd6104e2d98d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:14.143211 6902 tablet_service.cc:1467] Processing CreateTablet for tablet 629248af74c2468ba52fd6104e2d98d9 (DEFAULT_TABLE table=rebalance_test_table_12 [id=e0c6bd9b27794a2887eccea514c01058]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:14.143324 6771 tablet_service.cc:1467] Processing CreateTablet for tablet 94960e06ff44463285360fa7f10f1b68 (DEFAULT_TABLE table=rebalance_test_table_12 [id=e0c6bd9b27794a2887eccea514c01058]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:14.143532 6902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 629248af74c2468ba52fd6104e2d98d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:14.143573 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94960e06ff44463285360fa7f10f1b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:14.145555 6769 tablet_service.cc:1467] Processing CreateTablet for tablet 9890415b2d5a40a3b2f4434779ebf7eb (DEFAULT_TABLE table=rebalance_test_table_12 [id=e0c6bd9b27794a2887eccea514c01058]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:14.145669 8321 tablet_bootstrap.cc:492] T 629248af74c2468ba52fd6104e2d98d9 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:14.145802 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9890415b2d5a40a3b2f4434779ebf7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:14.146499 6509 tablet_service.cc:1467] Processing CreateTablet for tablet 94960e06ff44463285360fa7f10f1b68 (DEFAULT_TABLE table=rebalance_test_table_12 [id=e0c6bd9b27794a2887eccea514c01058]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:14.146582 8321 tablet_bootstrap.cc:654] T 629248af74c2468ba52fd6104e2d98d9 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:14.146730 6509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94960e06ff44463285360fa7f10f1b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:14.147162 6902 tablet_service.cc:1467] Processing CreateTablet for tablet 94960e06ff44463285360fa7f10f1b68 (DEFAULT_TABLE table=rebalance_test_table_12 [id=e0c6bd9b27794a2887eccea514c01058]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:14.147426 6902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94960e06ff44463285360fa7f10f1b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:14.148361 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 9890415b2d5a40a3b2f4434779ebf7eb (DEFAULT_TABLE table=rebalance_test_table_12 [id=e0c6bd9b27794a2887eccea514c01058]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:14.148587 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9890415b2d5a40a3b2f4434779ebf7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:14.148694 8320 tablet_bootstrap.cc:492] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:14.148764 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 9890415b2d5a40a3b2f4434779ebf7eb (DEFAULT_TABLE table=rebalance_test_table_12 [id=e0c6bd9b27794a2887eccea514c01058]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:14.148989 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9890415b2d5a40a3b2f4434779ebf7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:14.149878 8320 tablet_bootstrap.cc:654] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:14.149865 8322 tablet_bootstrap.cc:492] T 629248af74c2468ba52fd6104e2d98d9 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:14.150857 8322 tablet_bootstrap.cc:654] T 629248af74c2468ba52fd6104e2d98d9 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:14.151516 8320 tablet_bootstrap.cc:492] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:14.151521 8323 tablet_bootstrap.cc:492] T 94960e06ff44463285360fa7f10f1b68 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:14.151521 8321 tablet_bootstrap.cc:492] T 629248af74c2468ba52fd6104e2d98d9 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:14.151602 8321 ts_tablet_manager.cc:1367] T 629248af74c2468ba52fd6104e2d98d9 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:59:14.151608 8320 ts_tablet_manager.cc:1367] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:14.151952 8320 raft_consensus.cc:348] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:14.152029 8320 raft_consensus.cc:374] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:14.152050 8320 raft_consensus.cc:729] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:14.152046 8321 raft_consensus.cc:348] T 629248af74c2468ba52fd6104e2d98d9 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:14.152143 8321 raft_consensus.cc:374] T 629248af74c2468ba52fd6104e2d98d9 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:14.152166 8320 consensus_queue.cc:260] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:14.152189 8321 raft_consensus.cc:729] T 629248af74c2468ba52fd6104e2d98d9 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:14.152364 8323 tablet_bootstrap.cc:654] T 94960e06ff44463285360fa7f10f1b68 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:14.152326 8321 consensus_queue.cc:260] T 629248af74c2468ba52fd6104e2d98d9 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:14.152477 8320 ts_tablet_manager.cc:1398] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:14.152626 8321 ts_tablet_manager.cc:1398] T 629248af74c2468ba52fd6104e2d98d9 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:14.152659 8320 tablet_bootstrap.cc:492] T 94960e06ff44463285360fa7f10f1b68 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:14.152813 8321 tablet_bootstrap.cc:492] T 9890415b2d5a40a3b2f4434779ebf7eb P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:14.153517 8320 tablet_bootstrap.cc:654] T 94960e06ff44463285360fa7f10f1b68 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:14.153657 8321 tablet_bootstrap.cc:654] T 9890415b2d5a40a3b2f4434779ebf7eb P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:14.154703 8320 tablet_bootstrap.cc:492] T 94960e06ff44463285360fa7f10f1b68 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:14.154773 8320 ts_tablet_manager.cc:1367] T 94960e06ff44463285360fa7f10f1b68 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:14.155196 8320 raft_consensus.cc:348] T 94960e06ff44463285360fa7f10f1b68 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:14.155318 8320 raft_consensus.cc:374] T 94960e06ff44463285360fa7f10f1b68 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:14.155349 8320 raft_consensus.cc:729] T 94960e06ff44463285360fa7f10f1b68 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:14.155483 8320 consensus_queue.cc:260] T 94960e06ff44463285360fa7f10f1b68 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:14.155839 8320 ts_tablet_manager.cc:1398] T 94960e06ff44463285360fa7f10f1b68 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:14.155987 8320 tablet_bootstrap.cc:492] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:14.156874 8320 tablet_bootstrap.cc:654] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:14.157876 8322 tablet_bootstrap.cc:492] T 629248af74c2468ba52fd6104e2d98d9 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:14.157933 8320 tablet_bootstrap.cc:492] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:14.157963 8322 ts_tablet_manager.cc:1367] T 629248af74c2468ba52fd6104e2d98d9 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:59:14.158013 8320 ts_tablet_manager.cc:1367] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:14.158444 8320 raft_consensus.cc:348] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:14.158509 8320 raft_consensus.cc:374] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:14.158529 8320 raft_consensus.cc:729] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:14.158464 8322 raft_consensus.cc:348] T 629248af74c2468ba52fd6104e2d98d9 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:14.158562 8322 raft_consensus.cc:374] T 629248af74c2468ba52fd6104e2d98d9 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:14.158598 8322 raft_consensus.cc:729] T 629248af74c2468ba52fd6104e2d98d9 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:14.158619 8320 consensus_queue.cc:260] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:14.158749 8322 consensus_queue.cc:260] T 629248af74c2468ba52fd6104e2d98d9 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:14.158928 8320 ts_tablet_manager.cc:1398] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:14.159564 8321 tablet_bootstrap.cc:492] T 9890415b2d5a40a3b2f4434779ebf7eb P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:14.159616 8322 ts_tablet_manager.cc:1398] T 629248af74c2468ba52fd6104e2d98d9 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:14.159641 8321 ts_tablet_manager.cc:1367] T 9890415b2d5a40a3b2f4434779ebf7eb P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:59:14.159746 8323 tablet_bootstrap.cc:492] T 94960e06ff44463285360fa7f10f1b68 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:14.159770 8322 tablet_bootstrap.cc:492] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:14.159832 8323 ts_tablet_manager.cc:1367] T 94960e06ff44463285360fa7f10f1b68 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 13:59:14.160075 8321 raft_consensus.cc:348] T 9890415b2d5a40a3b2f4434779ebf7eb P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:14.160197 8321 raft_consensus.cc:374] T 9890415b2d5a40a3b2f4434779ebf7eb P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:14.160228 8321 raft_consensus.cc:729] T 9890415b2d5a40a3b2f4434779ebf7eb P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:14.160230 8323 raft_consensus.cc:348] T 94960e06ff44463285360fa7f10f1b68 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:14.160334 8323 raft_consensus.cc:374] T 94960e06ff44463285360fa7f10f1b68 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:14.160362 8323 raft_consensus.cc:729] T 94960e06ff44463285360fa7f10f1b68 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:14.160351 8321 consensus_queue.cc:260] T 9890415b2d5a40a3b2f4434779ebf7eb P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:14.160476 8323 consensus_queue.cc:260] T 94960e06ff44463285360fa7f10f1b68 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:14.160642 8322 tablet_bootstrap.cc:654] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:14.160634 8321 ts_tablet_manager.cc:1398] T 9890415b2d5a40a3b2f4434779ebf7eb P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:14.160722 8323 ts_tablet_manager.cc:1398] T 94960e06ff44463285360fa7f10f1b68 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:14.161782 8322 tablet_bootstrap.cc:492] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:14.161867 8322 ts_tablet_manager.cc:1367] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:14.162314 8322 raft_consensus.cc:348] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:14.162405 8322 raft_consensus.cc:374] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:14.162439 8322 raft_consensus.cc:729] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:14.162554 8322 consensus_queue.cc:260] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:14.162825 8322 ts_tablet_manager.cc:1398] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:14.164481 8323 tablet_bootstrap.cc:492] T 9890415b2d5a40a3b2f4434779ebf7eb P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:14.165513 8323 tablet_bootstrap.cc:654] T 9890415b2d5a40a3b2f4434779ebf7eb P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:14.166517 8323 tablet_bootstrap.cc:492] T 9890415b2d5a40a3b2f4434779ebf7eb P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:14.166586 8323 ts_tablet_manager.cc:1367] T 9890415b2d5a40a3b2f4434779ebf7eb P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:14.166958 8323 raft_consensus.cc:348] T 9890415b2d5a40a3b2f4434779ebf7eb P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:14.167045 8323 raft_consensus.cc:374] T 9890415b2d5a40a3b2f4434779ebf7eb P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:14.167079 8323 raft_consensus.cc:729] T 9890415b2d5a40a3b2f4434779ebf7eb P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:14.167208 8323 consensus_queue.cc:260] T 9890415b2d5a40a3b2f4434779ebf7eb P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:14.167616 8323 ts_tablet_manager.cc:1398] T 9890415b2d5a40a3b2f4434779ebf7eb P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:14.277137 8034 raft_consensus.cc:482] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:14.277266 8034 raft_consensus.cc:504] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:14.277722 8034 leader_election.cc:288] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:14.278017 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629248af74c2468ba52fd6104e2d98d9" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:14.278101 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629248af74c2468ba52fd6104e2d98d9" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:14.278155 6922 raft_consensus.cc:2454] T 629248af74c2468ba52fd6104e2d98d9 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:14.278241 6659 raft_consensus.cc:2454] T 629248af74c2468ba52fd6104e2d98d9 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:14.278420 6464 leader_election.cc:302] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:14.278580 8034 raft_consensus.cc:2781] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:14.278636 8034 raft_consensus.cc:482] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:14.278664 8034 raft_consensus.cc:3037] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:14.279553 8034 raft_consensus.cc:504] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:14.279848 8034 leader_election.cc:288] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:14.280155 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629248af74c2468ba52fd6104e2d98d9" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:14.280268 6659 raft_consensus.cc:3037] T 629248af74c2468ba52fd6104e2d98d9 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:14.280254 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629248af74c2468ba52fd6104e2d98d9" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:14.280355 6922 raft_consensus.cc:3037] T 629248af74c2468ba52fd6104e2d98d9 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:14.281210 6922 raft_consensus.cc:2454] T 629248af74c2468ba52fd6104e2d98d9 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:14.281210 6659 raft_consensus.cc:2454] T 629248af74c2468ba52fd6104e2d98d9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:14.281487 6465 leader_election.cc:302] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:14.281648 8034 raft_consensus.cc:2781] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:14.281730 8034 raft_consensus.cc:686] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Becoming Leader. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Running, Role: LEADER
I20240503 13:59:14.281847 8034 consensus_queue.cc:237] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:14.283540 6379 catalog_manager.cc:5484] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:14.444917 7535 raft_consensus.cc:482] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:14.445048 7535 raft_consensus.cc:504] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:14.445473 7535 leader_election.cc:288] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:14.445894 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94960e06ff44463285360fa7f10f1b68" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:14.445907 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94960e06ff44463285360fa7f10f1b68" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:14.446051 6791 raft_consensus.cc:2454] T 94960e06ff44463285360fa7f10f1b68 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:14.446050 6527 raft_consensus.cc:2454] T 94960e06ff44463285360fa7f10f1b68 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:14.446360 6856 leader_election.cc:302] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:14.446527 7535 raft_consensus.cc:2781] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:14.446592 7535 raft_consensus.cc:482] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:14.446625 7535 raft_consensus.cc:3037] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:14.447501 7535 raft_consensus.cc:504] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:14.447793 7535 leader_election.cc:288] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:14.448073 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94960e06ff44463285360fa7f10f1b68" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:14.448117 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94960e06ff44463285360fa7f10f1b68" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:14.448186 6791 raft_consensus.cc:3037] T 94960e06ff44463285360fa7f10f1b68 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:14.448251 6527 raft_consensus.cc:3037] T 94960e06ff44463285360fa7f10f1b68 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:14.448985 6791 raft_consensus.cc:2454] T 94960e06ff44463285360fa7f10f1b68 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:14.449020 6527 raft_consensus.cc:2454] T 94960e06ff44463285360fa7f10f1b68 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:14.449247 6856 leader_election.cc:302] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:14.449414 7535 raft_consensus.cc:2781] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:14.449540 7535 raft_consensus.cc:686] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:14.449733 7535 consensus_queue.cc:237] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:14.451628 6379 catalog_manager.cc:5484] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:14.462507 8034 raft_consensus.cc:482] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:14.462604 8034 raft_consensus.cc:504] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:14.462991 8034 leader_election.cc:288] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:14.463263 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9890415b2d5a40a3b2f4434779ebf7eb" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:14.463357 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9890415b2d5a40a3b2f4434779ebf7eb" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:14.463434 6791 raft_consensus.cc:2454] T 9890415b2d5a40a3b2f4434779ebf7eb P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:14.463492 6659 raft_consensus.cc:2454] T 9890415b2d5a40a3b2f4434779ebf7eb P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:14.463701 6463 leader_election.cc:302] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:14.463887 8034 raft_consensus.cc:2781] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:14.463945 8034 raft_consensus.cc:482] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:14.463994 8034 raft_consensus.cc:3037] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:14.464766 8034 raft_consensus.cc:504] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:14.465087 8034 leader_election.cc:288] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:14.465368 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9890415b2d5a40a3b2f4434779ebf7eb" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:14.465381 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9890415b2d5a40a3b2f4434779ebf7eb" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:14.465480 6791 raft_consensus.cc:3037] T 9890415b2d5a40a3b2f4434779ebf7eb P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:14.465484 6659 raft_consensus.cc:3037] T 9890415b2d5a40a3b2f4434779ebf7eb P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:14.466238 6659 raft_consensus.cc:2454] T 9890415b2d5a40a3b2f4434779ebf7eb P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:14.466408 6791 raft_consensus.cc:2454] T 9890415b2d5a40a3b2f4434779ebf7eb P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:14.466491 6465 leader_election.cc:302] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:14.466744 8182 raft_consensus.cc:2781] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:14.466975 8182 raft_consensus.cc:686] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Becoming Leader. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Running, Role: LEADER
I20240503 13:59:14.467135 8182 consensus_queue.cc:237] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:14.469067 6379 catalog_manager.cc:5484] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:14.627909 8338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:14.628124 8338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:14.631322 8338 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_12'
I20240503 13:59:14.680560 8336 consensus_queue.cc:1035] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:14.683712 8182 consensus_queue.cc:1035] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:14.722005 6659 raft_consensus.cc:1261] T 9890415b2d5a40a3b2f4434779ebf7eb P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:14.722431 8034 consensus_queue.cc:1035] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:14.722916 6791 raft_consensus.cc:1261] T 9890415b2d5a40a3b2f4434779ebf7eb P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:14.723462 6527 raft_consensus.cc:1261] T 94960e06ff44463285360fa7f10f1b68 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:14.723536 8034 consensus_queue.cc:1035] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:14.725883 8297 consensus_queue.cc:1035] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:14.727419 6789 raft_consensus.cc:1261] T 94960e06ff44463285360fa7f10f1b68 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:14.728408 7535 consensus_queue.cc:1035] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:14.733554 8362 mvcc.cc:205] Tried to move back new op lower bound from 7023594515336249344 to 7023594514298605568. Current Snapshot: MvccSnapshot[applied={T|T < 7023594515336249344}]
I20240503 13:59:14.735797 8363 mvcc.cc:205] Tried to move back new op lower bound from 7023594515336249344 to 7023594514298605568. Current Snapshot: MvccSnapshot[applied={T|T < 7023594515336249344}]
I20240503 13:59:14.741359 8365 mvcc.cc:205] Tried to move back new op lower bound from 7023594515340742656 to 7023594514227134464. Current Snapshot: MvccSnapshot[applied={T|T < 7023594515340742656}]
I20240503 13:59:14.747820 8373 mvcc.cc:205] Tried to move back new op lower bound from 7023594515336249344 to 7023594514298605568. Current Snapshot: MvccSnapshot[applied={T|T < 7023594515336249344}]
INSERT report
rows total: 2000
time total: 115.506 ms
time per row: 0.057753 ms
I20240503 13:59:14.791679 6379 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:14.797062 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 752e0d08505b4c8cb4b2a44cc00ce858 (DEFAULT_TABLE table=rebalance_test_table_13 [id=37a0688b297f4755a4cc7c147a101e22]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:14.797194 6902 tablet_service.cc:1467] Processing CreateTablet for tablet 752e0d08505b4c8cb4b2a44cc00ce858 (DEFAULT_TABLE table=rebalance_test_table_13 [id=37a0688b297f4755a4cc7c147a101e22]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:14.797237 6769 tablet_service.cc:1467] Processing CreateTablet for tablet 752e0d08505b4c8cb4b2a44cc00ce858 (DEFAULT_TABLE table=rebalance_test_table_13 [id=37a0688b297f4755a4cc7c147a101e22]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:14.797394 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752e0d08505b4c8cb4b2a44cc00ce858. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:14.797439 6902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752e0d08505b4c8cb4b2a44cc00ce858. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:14.797552 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752e0d08505b4c8cb4b2a44cc00ce858. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:14.798617 6900 tablet_service.cc:1467] Processing CreateTablet for tablet 586369c0c3174070a3e184af38b3adf7 (DEFAULT_TABLE table=rebalance_test_table_13 [id=37a0688b297f4755a4cc7c147a101e22]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:14.798877 6900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 586369c0c3174070a3e184af38b3adf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:14.800299 6639 tablet_service.cc:1467] Processing CreateTablet for tablet da69140936424a31b66c87e2ad4092cb (DEFAULT_TABLE table=rebalance_test_table_13 [id=37a0688b297f4755a4cc7c147a101e22]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:14.800297 6508 tablet_service.cc:1467] Processing CreateTablet for tablet da69140936424a31b66c87e2ad4092cb (DEFAULT_TABLE table=rebalance_test_table_13 [id=37a0688b297f4755a4cc7c147a101e22]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:14.800549 6639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da69140936424a31b66c87e2ad4092cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:14.800572 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da69140936424a31b66c87e2ad4092cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:14.800649 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 586369c0c3174070a3e184af38b3adf7 (DEFAULT_TABLE table=rebalance_test_table_13 [id=37a0688b297f4755a4cc7c147a101e22]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:14.800843 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 586369c0c3174070a3e184af38b3adf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:14.800916 6901 tablet_service.cc:1467] Processing CreateTablet for tablet da69140936424a31b66c87e2ad4092cb (DEFAULT_TABLE table=rebalance_test_table_13 [id=37a0688b297f4755a4cc7c147a101e22]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:14.801112 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da69140936424a31b66c87e2ad4092cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:14.803040 8396 tablet_bootstrap.cc:492] T 752e0d08505b4c8cb4b2a44cc00ce858 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:14.803172 6769 tablet_service.cc:1467] Processing CreateTablet for tablet 586369c0c3174070a3e184af38b3adf7 (DEFAULT_TABLE table=rebalance_test_table_13 [id=37a0688b297f4755a4cc7c147a101e22]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:14.803422 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 586369c0c3174070a3e184af38b3adf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:14.804205 8397 tablet_bootstrap.cc:492] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:14.804879 8398 tablet_bootstrap.cc:492] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:14.805104 8397 tablet_bootstrap.cc:654] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:14.805467 8396 tablet_bootstrap.cc:654] T 752e0d08505b4c8cb4b2a44cc00ce858 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:14.806416 8397 tablet_bootstrap.cc:492] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:14.806509 8397 ts_tablet_manager.cc:1367] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:14.806514 8398 tablet_bootstrap.cc:654] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:14.806635 8395 tablet_bootstrap.cc:492] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:14.806874 8396 tablet_bootstrap.cc:492] T 752e0d08505b4c8cb4b2a44cc00ce858 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:14.807009 8396 ts_tablet_manager.cc:1367] T 752e0d08505b4c8cb4b2a44cc00ce858 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:14.806977 8397 raft_consensus.cc:348] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:14.807135 8397 raft_consensus.cc:374] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:14.807186 8397 raft_consensus.cc:729] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:14.807368 8397 consensus_queue.cc:260] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:14.807667 8397 ts_tablet_manager.cc:1398] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:14.807716 8398 tablet_bootstrap.cc:492] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:14.807792 8398 ts_tablet_manager.cc:1367] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:14.807842 8397 tablet_bootstrap.cc:492] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:14.807983 8396 raft_consensus.cc:348] T 752e0d08505b4c8cb4b2a44cc00ce858 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:14.808102 8396 raft_consensus.cc:374] T 752e0d08505b4c8cb4b2a44cc00ce858 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:14.808152 8396 raft_consensus.cc:729] T 752e0d08505b4c8cb4b2a44cc00ce858 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:14.808238 8398 raft_consensus.cc:348] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:14.808327 8398 raft_consensus.cc:374] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:14.808359 8398 raft_consensus.cc:729] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:14.808490 8398 consensus_queue.cc:260] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:14.808487 8396 consensus_queue.cc:260] T 752e0d08505b4c8cb4b2a44cc00ce858 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:14.808758 8397 tablet_bootstrap.cc:654] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:14.808765 8398 ts_tablet_manager.cc:1398] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:14.808769 8396 ts_tablet_manager.cc:1398] T 752e0d08505b4c8cb4b2a44cc00ce858 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:14.809695 8397 tablet_bootstrap.cc:492] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:14.809764 8397 ts_tablet_manager.cc:1367] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:14.809688 8396 tablet_bootstrap.cc:492] T 586369c0c3174070a3e184af38b3adf7 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:14.810164 8397 raft_consensus.cc:348] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:14.810253 8397 raft_consensus.cc:374] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:14.810282 8397 raft_consensus.cc:729] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:14.810377 8397 consensus_queue.cc:260] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:14.810532 8396 tablet_bootstrap.cc:654] T 586369c0c3174070a3e184af38b3adf7 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:14.810595 8397 ts_tablet_manager.cc:1398] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:14.811555 8396 tablet_bootstrap.cc:492] T 586369c0c3174070a3e184af38b3adf7 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:14.811633 8396 ts_tablet_manager.cc:1367] T 586369c0c3174070a3e184af38b3adf7 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:14.812036 8396 raft_consensus.cc:348] T 586369c0c3174070a3e184af38b3adf7 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:14.812144 8396 raft_consensus.cc:374] T 586369c0c3174070a3e184af38b3adf7 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:14.812175 8396 raft_consensus.cc:729] T 586369c0c3174070a3e184af38b3adf7 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:14.812711 8395 tablet_bootstrap.cc:654] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:14.812453 8396 consensus_queue.cc:260] T 586369c0c3174070a3e184af38b3adf7 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:14.813266 8396 ts_tablet_manager.cc:1398] T 586369c0c3174070a3e184af38b3adf7 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:14.813426 8396 tablet_bootstrap.cc:492] T da69140936424a31b66c87e2ad4092cb P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:14.814527 8396 tablet_bootstrap.cc:654] T da69140936424a31b66c87e2ad4092cb P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:14.815891 8396 tablet_bootstrap.cc:492] T da69140936424a31b66c87e2ad4092cb P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:14.815965 8396 ts_tablet_manager.cc:1367] T da69140936424a31b66c87e2ad4092cb P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:14.816303 8396 raft_consensus.cc:348] T da69140936424a31b66c87e2ad4092cb P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:14.816385 8396 raft_consensus.cc:374] T da69140936424a31b66c87e2ad4092cb P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:14.816407 8396 raft_consensus.cc:729] T da69140936424a31b66c87e2ad4092cb P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:14.816545 8396 consensus_queue.cc:260] T da69140936424a31b66c87e2ad4092cb P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:14.816835 8396 ts_tablet_manager.cc:1398] T da69140936424a31b66c87e2ad4092cb P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:14.824085 8394 raft_consensus.cc:482] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:14.824106 8395 tablet_bootstrap.cc:492] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:14.824211 8395 ts_tablet_manager.cc:1367] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240503 13:59:14.824184 8394 raft_consensus.cc:504] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:14.824577 8394 leader_election.cc:288] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:14.824661 8395 raft_consensus.cc:348] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:14.824769 8395 raft_consensus.cc:374] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:14.824807 8395 raft_consensus.cc:729] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:14.824853 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752e0d08505b4c8cb4b2a44cc00ce858" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:14.824939 8395 consensus_queue.cc:260] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:14.824982 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752e0d08505b4c8cb4b2a44cc00ce858" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:14.825024 6660 raft_consensus.cc:2454] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:14.825076 6922 raft_consensus.cc:2454] T 752e0d08505b4c8cb4b2a44cc00ce858 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:14.825281 8395 ts_tablet_manager.cc:1398] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:14.825357 6727 leader_election.cc:302] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:14.825508 8394 raft_consensus.cc:2781] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:14.825533 8395 tablet_bootstrap.cc:492] T da69140936424a31b66c87e2ad4092cb P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:14.825562 8394 raft_consensus.cc:482] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:14.825599 8394 raft_consensus.cc:3037] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:14.826426 8395 tablet_bootstrap.cc:654] T da69140936424a31b66c87e2ad4092cb P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:14.826450 8394 raft_consensus.cc:504] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:14.826795 8394 leader_election.cc:288] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:14.826985 8336 raft_consensus.cc:482] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:14.827066 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752e0d08505b4c8cb4b2a44cc00ce858" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:14.827085 8336 raft_consensus.cc:504] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:14.827159 6660 raft_consensus.cc:3037] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:14.827500 8336 leader_election.cc:288] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:14.827751 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da69140936424a31b66c87e2ad4092cb" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:14.827895 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da69140936424a31b66c87e2ad4092cb" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:14.828012 6922 raft_consensus.cc:2454] T da69140936424a31b66c87e2ad4092cb P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:14.828102 6660 raft_consensus.cc:2454] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
W20240503 13:59:14.828246 6465 leader_election.cc:341] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:59:14.828356 6464 leader_election.cc:302] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters: 2145d15ab2d74c0287839266f1411a43
I20240503 13:59:14.828440 6727 leader_election.cc:302] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:14.828565 8336 raft_consensus.cc:2781] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:14.828603 8336 raft_consensus.cc:482] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:14.828634 8336 raft_consensus.cc:3037] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:14.829071 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752e0d08505b4c8cb4b2a44cc00ce858" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:14.829185 6922 raft_consensus.cc:3037] T 752e0d08505b4c8cb4b2a44cc00ce858 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:14.829470 8336 raft_consensus.cc:504] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:14.829774 8336 leader_election.cc:288] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Requested vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:14.830013 6922 raft_consensus.cc:2454] T 752e0d08505b4c8cb4b2a44cc00ce858 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:14.830018 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da69140936424a31b66c87e2ad4092cb" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:14.830183 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da69140936424a31b66c87e2ad4092cb" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:14.830278 6921 raft_consensus.cc:3037] T da69140936424a31b66c87e2ad4092cb P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
W20240503 13:59:14.830317 6465 leader_election.cc:341] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:59:14.830461 8394 raft_consensus.cc:2781] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:14.830556 8394 raft_consensus.cc:686] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:14.830679 8394 consensus_queue.cc:237] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:14.831089 6921 raft_consensus.cc:2454] T da69140936424a31b66c87e2ad4092cb P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:14.831897 6464 leader_election.cc:302] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters: 2145d15ab2d74c0287839266f1411a43
I20240503 13:59:14.832110 8336 raft_consensus.cc:2781] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:14.832183 8336 raft_consensus.cc:686] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Becoming Leader. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Running, Role: LEADER
I20240503 13:59:14.832298 8336 consensus_queue.cc:237] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:14.833508 6379 catalog_manager.cc:5484] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:14.833600 8395 tablet_bootstrap.cc:492] T da69140936424a31b66c87e2ad4092cb P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:14.833685 8395 ts_tablet_manager.cc:1367] T da69140936424a31b66c87e2ad4092cb P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 13:59:14.834149 8395 raft_consensus.cc:348] T da69140936424a31b66c87e2ad4092cb P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:14.834242 8395 raft_consensus.cc:374] T da69140936424a31b66c87e2ad4092cb P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:14.834136 6378 catalog_manager.cc:5484] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:14.834271 8395 raft_consensus.cc:729] T da69140936424a31b66c87e2ad4092cb P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:14.834398 8395 consensus_queue.cc:260] T da69140936424a31b66c87e2ad4092cb P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:14.834668 8395 ts_tablet_manager.cc:1398] T da69140936424a31b66c87e2ad4092cb P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:14.834832 8395 tablet_bootstrap.cc:492] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:14.835733 8395 tablet_bootstrap.cc:654] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:14.837170 8395 tablet_bootstrap.cc:492] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:14.837298 8395 ts_tablet_manager.cc:1367] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:14.837795 8395 raft_consensus.cc:348] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:14.837955 8395 raft_consensus.cc:374] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:14.838035 8395 raft_consensus.cc:729] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:14.838169 8395 consensus_queue.cc:260] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:14.838485 8395 ts_tablet_manager.cc:1398] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:15.060868 8394 raft_consensus.cc:482] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:15.060988 8394 raft_consensus.cc:504] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:15.061409 8394 leader_election.cc:288] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:15.061715 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "586369c0c3174070a3e184af38b3adf7" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:15.061798 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "586369c0c3174070a3e184af38b3adf7" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:15.061864 6921 raft_consensus.cc:2454] T 586369c0c3174070a3e184af38b3adf7 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:15.061945 6660 raft_consensus.cc:2454] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:15.062192 6726 leader_election.cc:302] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:15.062407 8394 raft_consensus.cc:2781] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:15.062474 8394 raft_consensus.cc:482] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:15.062515 8394 raft_consensus.cc:3037] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:15.063402 8394 raft_consensus.cc:504] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:15.063767 8394 leader_election.cc:288] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:15.064107 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "586369c0c3174070a3e184af38b3adf7" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:15.064117 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "586369c0c3174070a3e184af38b3adf7" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:15.064211 6660 raft_consensus.cc:3037] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:15.064224 6921 raft_consensus.cc:3037] T 586369c0c3174070a3e184af38b3adf7 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:15.064994 6660 raft_consensus.cc:2454] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:15.064998 6921 raft_consensus.cc:2454] T 586369c0c3174070a3e184af38b3adf7 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:15.065292 6726 leader_election.cc:302] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:15.065474 8394 raft_consensus.cc:2781] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:15.065567 8394 raft_consensus.cc:686] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:15.065706 8394 consensus_queue.cc:237] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:15.067567 6379 catalog_manager.cc:5484] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:15.154167 8333 raft_consensus.cc:482] T da69140936424a31b66c87e2ad4092cb P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:15.154361 8333 raft_consensus.cc:504] T da69140936424a31b66c87e2ad4092cb P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:15.154783 8333 leader_election.cc:288] T da69140936424a31b66c87e2ad4092cb P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:15.155113 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da69140936424a31b66c87e2ad4092cb" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:15.155233 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da69140936424a31b66c87e2ad4092cb" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:15.155402 6921 raft_consensus.cc:2379] T da69140936424a31b66c87e2ad4092cb P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2145d15ab2d74c0287839266f1411a43 in current term 1: Already voted for candidate 5f46e19644fd4f4b825c86ba6af768da in this term.
I20240503 13:59:15.155647 6595 leader_election.cc:302] T da69140936424a31b66c87e2ad4092cb P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43; no voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460
I20240503 13:59:15.155866 8333 raft_consensus.cc:3037] T da69140936424a31b66c87e2ad4092cb P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:15.156816 8333 raft_consensus.cc:2726] T da69140936424a31b66c87e2ad4092cb P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 13:59:15.205071 8412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:15.205338 8412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:15.208537 8412 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 13:59:15.212252 8394 consensus_queue.cc:1035] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:15.218086 8417 consensus_queue.cc:1035] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_13'
I20240503 13:59:15.289781 8182 consensus_queue.cc:1035] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:15.293170 6922 raft_consensus.cc:1261] T da69140936424a31b66c87e2ad4092cb P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:15.293697 8182 consensus_queue.cc:1035] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:15.295504 6920 raft_consensus.cc:1261] T 586369c0c3174070a3e184af38b3adf7 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:15.295754 6658 raft_consensus.cc:1261] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:15.298172 8417 consensus_queue.cc:1035] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:15.300583 8454 consensus_queue.cc:1035] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 109.007 ms
time per row: 0.0545037 ms
I20240503 13:59:15.362655 6386 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:15.367934 6640 tablet_service.cc:1467] Processing CreateTablet for tablet b22f2b5852fb48a5a5400f843de82702 (DEFAULT_TABLE table=rebalance_test_table_14 [id=2593391fce974113886702729b8e3d24]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:15.368121 6769 tablet_service.cc:1467] Processing CreateTablet for tablet b22f2b5852fb48a5a5400f843de82702 (DEFAULT_TABLE table=rebalance_test_table_14 [id=2593391fce974113886702729b8e3d24]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:15.368283 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b22f2b5852fb48a5a5400f843de82702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:15.368428 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b22f2b5852fb48a5a5400f843de82702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:15.368906 6639 tablet_service.cc:1467] Processing CreateTablet for tablet ba1a0a1ecff746d9ace73f0c2b75d991 (DEFAULT_TABLE table=rebalance_test_table_14 [id=2593391fce974113886702729b8e3d24]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:15.369140 6639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1a0a1ecff746d9ace73f0c2b75d991. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:15.370256 8467 tablet_bootstrap.cc:492] T b22f2b5852fb48a5a5400f843de82702 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:15.370703 6771 tablet_service.cc:1467] Processing CreateTablet for tablet ba1a0a1ecff746d9ace73f0c2b75d991 (DEFAULT_TABLE table=rebalance_test_table_14 [id=2593391fce974113886702729b8e3d24]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:15.370925 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1a0a1ecff746d9ace73f0c2b75d991. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:15.371179 8467 tablet_bootstrap.cc:654] T b22f2b5852fb48a5a5400f843de82702 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:15.371198 8468 tablet_bootstrap.cc:492] T b22f2b5852fb48a5a5400f843de82702 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:15.371887 8468 tablet_bootstrap.cc:654] T b22f2b5852fb48a5a5400f843de82702 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:15.372321 8467 tablet_bootstrap.cc:492] T b22f2b5852fb48a5a5400f843de82702 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:15.372392 8467 ts_tablet_manager.cc:1367] T b22f2b5852fb48a5a5400f843de82702 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:15.372736 8468 tablet_bootstrap.cc:492] T b22f2b5852fb48a5a5400f843de82702 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:15.372823 8467 raft_consensus.cc:348] T b22f2b5852fb48a5a5400f843de82702 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:15.372931 8467 raft_consensus.cc:374] T b22f2b5852fb48a5a5400f843de82702 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:15.372931 8468 ts_tablet_manager.cc:1367] T b22f2b5852fb48a5a5400f843de82702 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:15.372969 8467 raft_consensus.cc:729] T b22f2b5852fb48a5a5400f843de82702 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:15.373095 8467 consensus_queue.cc:260] T b22f2b5852fb48a5a5400f843de82702 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:15.373594 6508 tablet_service.cc:1467] Processing CreateTablet for tablet b22f2b5852fb48a5a5400f843de82702 (DEFAULT_TABLE table=rebalance_test_table_14 [id=2593391fce974113886702729b8e3d24]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:15.373714 6507 tablet_service.cc:1467] Processing CreateTablet for tablet ba1a0a1ecff746d9ace73f0c2b75d991 (DEFAULT_TABLE table=rebalance_test_table_14 [id=2593391fce974113886702729b8e3d24]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:15.373868 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b22f2b5852fb48a5a5400f843de82702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:15.373926 8468 raft_consensus.cc:348] T b22f2b5852fb48a5a5400f843de82702 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:15.374014 8468 raft_consensus.cc:374] T b22f2b5852fb48a5a5400f843de82702 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:15.374050 8468 raft_consensus.cc:729] T b22f2b5852fb48a5a5400f843de82702 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:15.374060 6901 tablet_service.cc:1467] Processing CreateTablet for tablet 34ede4b3eb8740a9922a061569f6e663 (DEFAULT_TABLE table=rebalance_test_table_14 [id=2593391fce974113886702729b8e3d24]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:15.374182 8468 consensus_queue.cc:260] T b22f2b5852fb48a5a5400f843de82702 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:15.374274 6507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1a0a1ecff746d9ace73f0c2b75d991. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:15.374300 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34ede4b3eb8740a9922a061569f6e663. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:15.374538 8468 ts_tablet_manager.cc:1398] T b22f2b5852fb48a5a5400f843de82702 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:15.374678 8468 tablet_bootstrap.cc:492] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:15.375499 8468 tablet_bootstrap.cc:654] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:15.373594 6509 tablet_service.cc:1467] Processing CreateTablet for tablet 34ede4b3eb8740a9922a061569f6e663 (DEFAULT_TABLE table=rebalance_test_table_14 [id=2593391fce974113886702729b8e3d24]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:15.376209 6509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34ede4b3eb8740a9922a061569f6e663. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:15.376494 8473 tablet_bootstrap.cc:492] T 34ede4b3eb8740a9922a061569f6e663 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:15.376952 8468 tablet_bootstrap.cc:492] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:15.377029 8468 ts_tablet_manager.cc:1367] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:15.377133 8467 ts_tablet_manager.cc:1398] T b22f2b5852fb48a5a5400f843de82702 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240503 13:59:15.377313 8467 tablet_bootstrap.cc:492] T ba1a0a1ecff746d9ace73f0c2b75d991 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:15.377461 8468 raft_consensus.cc:348] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:15.377568 8468 raft_consensus.cc:374] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:15.377597 8468 raft_consensus.cc:729] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:15.377713 8468 consensus_queue.cc:260] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:15.378055 8468 ts_tablet_manager.cc:1398] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:15.378149 8467 tablet_bootstrap.cc:654] T ba1a0a1ecff746d9ace73f0c2b75d991 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:15.379849 6769 tablet_service.cc:1467] Processing CreateTablet for tablet 34ede4b3eb8740a9922a061569f6e663 (DEFAULT_TABLE table=rebalance_test_table_14 [id=2593391fce974113886702729b8e3d24]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:15.380079 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34ede4b3eb8740a9922a061569f6e663. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:15.380098 8473 tablet_bootstrap.cc:654] T 34ede4b3eb8740a9922a061569f6e663 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:15.380146 8467 tablet_bootstrap.cc:492] T ba1a0a1ecff746d9ace73f0c2b75d991 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:15.380204 8467 ts_tablet_manager.cc:1367] T ba1a0a1ecff746d9ace73f0c2b75d991 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:59:15.380570 8467 raft_consensus.cc:348] T ba1a0a1ecff746d9ace73f0c2b75d991 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:15.380649 8467 raft_consensus.cc:374] T ba1a0a1ecff746d9ace73f0c2b75d991 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:15.380676 8467 raft_consensus.cc:729] T ba1a0a1ecff746d9ace73f0c2b75d991 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:15.380779 8467 consensus_queue.cc:260] T ba1a0a1ecff746d9ace73f0c2b75d991 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:15.381079 8467 ts_tablet_manager.cc:1398] T ba1a0a1ecff746d9ace73f0c2b75d991 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:15.381160 8473 tablet_bootstrap.cc:492] T 34ede4b3eb8740a9922a061569f6e663 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:15.381253 8473 ts_tablet_manager.cc:1367] T 34ede4b3eb8740a9922a061569f6e663 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:15.381696 8473 raft_consensus.cc:348] T 34ede4b3eb8740a9922a061569f6e663 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:15.381801 8473 raft_consensus.cc:374] T 34ede4b3eb8740a9922a061569f6e663 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:15.381836 8473 raft_consensus.cc:729] T 34ede4b3eb8740a9922a061569f6e663 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:15.381949 8473 consensus_queue.cc:260] T 34ede4b3eb8740a9922a061569f6e663 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:15.382004 8467 tablet_bootstrap.cc:492] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:15.382241 8473 ts_tablet_manager.cc:1398] T 34ede4b3eb8740a9922a061569f6e663 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:15.382819 8467 tablet_bootstrap.cc:654] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:15.383651 8471 tablet_bootstrap.cc:492] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:15.384522 8467 tablet_bootstrap.cc:492] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:15.384594 8471 tablet_bootstrap.cc:654] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:15.384605 8467 ts_tablet_manager.cc:1367] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:59:15.385006 8467 raft_consensus.cc:348] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:15.385126 8467 raft_consensus.cc:374] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:15.385162 8467 raft_consensus.cc:729] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:15.385280 8467 consensus_queue.cc:260] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:15.385540 8467 ts_tablet_manager.cc:1398] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:15.386142 8471 tablet_bootstrap.cc:492] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:15.386229 8471 ts_tablet_manager.cc:1367] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:15.386584 8471 raft_consensus.cc:348] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:15.386672 8471 raft_consensus.cc:374] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:15.386709 8471 raft_consensus.cc:729] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:15.386834 8471 consensus_queue.cc:260] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:15.387075 8471 ts_tablet_manager.cc:1398] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:15.387228 8471 tablet_bootstrap.cc:492] T ba1a0a1ecff746d9ace73f0c2b75d991 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:15.388008 8471 tablet_bootstrap.cc:654] T ba1a0a1ecff746d9ace73f0c2b75d991 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:15.389014 8471 tablet_bootstrap.cc:492] T ba1a0a1ecff746d9ace73f0c2b75d991 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:15.389212 8471 ts_tablet_manager.cc:1367] T ba1a0a1ecff746d9ace73f0c2b75d991 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:15.389598 8471 raft_consensus.cc:348] T ba1a0a1ecff746d9ace73f0c2b75d991 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:15.389705 8471 raft_consensus.cc:374] T ba1a0a1ecff746d9ace73f0c2b75d991 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:15.389735 8471 raft_consensus.cc:729] T ba1a0a1ecff746d9ace73f0c2b75d991 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:15.389843 8471 consensus_queue.cc:260] T ba1a0a1ecff746d9ace73f0c2b75d991 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:15.390105 8471 ts_tablet_manager.cc:1398] T ba1a0a1ecff746d9ace73f0c2b75d991 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:15.390267 8471 tablet_bootstrap.cc:492] T 34ede4b3eb8740a9922a061569f6e663 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:15.390903 8471 tablet_bootstrap.cc:654] T 34ede4b3eb8740a9922a061569f6e663 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:15.391885 8471 tablet_bootstrap.cc:492] T 34ede4b3eb8740a9922a061569f6e663 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:15.391949 8471 ts_tablet_manager.cc:1367] T 34ede4b3eb8740a9922a061569f6e663 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:15.392395 8471 raft_consensus.cc:348] T 34ede4b3eb8740a9922a061569f6e663 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:15.392469 8471 raft_consensus.cc:374] T 34ede4b3eb8740a9922a061569f6e663 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:15.392490 8471 raft_consensus.cc:729] T 34ede4b3eb8740a9922a061569f6e663 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:15.392582 8471 consensus_queue.cc:260] T 34ede4b3eb8740a9922a061569f6e663 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:15.392809 8471 ts_tablet_manager.cc:1398] T 34ede4b3eb8740a9922a061569f6e663 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:15.503319 8423 raft_consensus.cc:482] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:15.503456 8423 raft_consensus.cc:504] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:15.503858 8423 leader_election.cc:288] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:15.504235 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1a0a1ecff746d9ace73f0c2b75d991" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:15.504249 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1a0a1ecff746d9ace73f0c2b75d991" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:15.504372 6527 raft_consensus.cc:2454] T ba1a0a1ecff746d9ace73f0c2b75d991 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:15.504391 6790 raft_consensus.cc:2454] T ba1a0a1ecff746d9ace73f0c2b75d991 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:15.504644 6595 leader_election.cc:302] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:15.504833 8423 raft_consensus.cc:2781] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:15.504890 8423 raft_consensus.cc:482] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:15.504928 8423 raft_consensus.cc:3037] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:15.505829 8423 raft_consensus.cc:504] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:15.506141 8423 leader_election.cc:288] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:15.506420 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1a0a1ecff746d9ace73f0c2b75d991" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:15.506443 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1a0a1ecff746d9ace73f0c2b75d991" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:15.506525 6527 raft_consensus.cc:3037] T ba1a0a1ecff746d9ace73f0c2b75d991 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:15.506568 6790 raft_consensus.cc:3037] T ba1a0a1ecff746d9ace73f0c2b75d991 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:15.507350 6790 raft_consensus.cc:2454] T ba1a0a1ecff746d9ace73f0c2b75d991 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:15.507364 6527 raft_consensus.cc:2454] T ba1a0a1ecff746d9ace73f0c2b75d991 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:15.507627 6594 leader_election.cc:302] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:15.507794 8423 raft_consensus.cc:2781] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:15.507884 8423 raft_consensus.cc:686] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 13:59:15.508069 8423 consensus_queue.cc:237] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:15.510107 6386 catalog_manager.cc:5484] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:15.577682 8466 raft_consensus.cc:482] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:15.577787 8466 raft_consensus.cc:504] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:15.578159 8466 leader_election.cc:288] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:15.578478 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b22f2b5852fb48a5a5400f843de82702" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:15.578508 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b22f2b5852fb48a5a5400f843de82702" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:15.578630 6790 raft_consensus.cc:2454] T b22f2b5852fb48a5a5400f843de82702 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:15.578629 6660 raft_consensus.cc:2454] T b22f2b5852fb48a5a5400f843de82702 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:15.578912 6465 leader_election.cc:302] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:15.579116 8466 raft_consensus.cc:2781] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:15.579182 8466 raft_consensus.cc:482] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:15.579212 8466 raft_consensus.cc:3037] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:15.580107 8466 raft_consensus.cc:504] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:15.580457 8466 leader_election.cc:288] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Requested vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:15.580719 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b22f2b5852fb48a5a5400f843de82702" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:15.580768 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b22f2b5852fb48a5a5400f843de82702" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:15.580821 6660 raft_consensus.cc:3037] T b22f2b5852fb48a5a5400f843de82702 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:15.580896 6790 raft_consensus.cc:3037] T b22f2b5852fb48a5a5400f843de82702 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:15.581557 6660 raft_consensus.cc:2454] T b22f2b5852fb48a5a5400f843de82702 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:15.581615 6790 raft_consensus.cc:2454] T b22f2b5852fb48a5a5400f843de82702 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:15.581883 6465 leader_election.cc:302] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:15.582067 8466 raft_consensus.cc:2781] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:15.582154 8466 raft_consensus.cc:686] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Becoming Leader. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Running, Role: LEADER
I20240503 13:59:15.582279 8466 consensus_queue.cc:237] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:15.584224 6386 catalog_manager.cc:5484] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:15.602700 8454 raft_consensus.cc:482] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:15.602828 8454 raft_consensus.cc:504] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:15.603173 8454 leader_election.cc:288] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:15.603545 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34ede4b3eb8740a9922a061569f6e663" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:15.603592 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34ede4b3eb8740a9922a061569f6e663" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:15.603708 6527 raft_consensus.cc:2454] T 34ede4b3eb8740a9922a061569f6e663 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:15.603744 6922 raft_consensus.cc:2454] T 34ede4b3eb8740a9922a061569f6e663 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:15.603989 6726 leader_election.cc:302] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:15.604174 8454 raft_consensus.cc:2781] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:15.604238 8454 raft_consensus.cc:482] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:15.604264 8454 raft_consensus.cc:3037] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:15.605125 8454 raft_consensus.cc:504] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:15.605417 8454 leader_election.cc:288] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:15.605759 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34ede4b3eb8740a9922a061569f6e663" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:15.605780 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34ede4b3eb8740a9922a061569f6e663" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:15.605888 6922 raft_consensus.cc:3037] T 34ede4b3eb8740a9922a061569f6e663 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:15.605888 6527 raft_consensus.cc:3037] T 34ede4b3eb8740a9922a061569f6e663 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:15.606819 6527 raft_consensus.cc:2454] T 34ede4b3eb8740a9922a061569f6e663 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:15.606833 6922 raft_consensus.cc:2454] T 34ede4b3eb8740a9922a061569f6e663 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:15.607110 6726 leader_election.cc:302] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:15.607266 8454 raft_consensus.cc:2781] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:15.607379 8454 raft_consensus.cc:686] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:15.607525 8454 consensus_queue.cc:237] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:15.609462 6386 catalog_manager.cc:5484] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:15.776444 8483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:15.776716 8483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:15.779865 8483 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_14'
I20240503 13:59:15.894804 6790 raft_consensus.cc:1261] T b22f2b5852fb48a5a5400f843de82702 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:15.895800 8466 consensus_queue.cc:1035] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:15.895918 6790 raft_consensus.cc:1261] T ba1a0a1ecff746d9ace73f0c2b75d991 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:15.895941 6660 raft_consensus.cc:1261] T b22f2b5852fb48a5a5400f843de82702 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:15.896327 8466 consensus_queue.cc:1035] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:15.896759 6527 raft_consensus.cc:1261] T ba1a0a1ecff746d9ace73f0c2b75d991 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:15.897559 8333 consensus_queue.cc:1035] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:15.897666 8423 consensus_queue.cc:1035] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:15.901929 6921 raft_consensus.cc:1261] T 34ede4b3eb8740a9922a061569f6e663 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:15.902894 6527 raft_consensus.cc:1261] T 34ede4b3eb8740a9922a061569f6e663 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:15.903096 8454 consensus_queue.cc:1035] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:15.903859 8394 consensus_queue.cc:1035] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 143.092 ms
time per row: 0.0715458 ms
I20240503 13:59:15.967398 6379 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:15.972530 6901 tablet_service.cc:1467] Processing CreateTablet for tablet 400c54bdecfa44e08c7eca046bcd2ffc (DEFAULT_TABLE table=rebalance_test_table_15 [id=65cfc473fc21464c991e7a62f281b04e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:15.972796 6639 tablet_service.cc:1467] Processing CreateTablet for tablet 400c54bdecfa44e08c7eca046bcd2ffc (DEFAULT_TABLE table=rebalance_test_table_15 [id=65cfc473fc21464c991e7a62f281b04e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:15.972877 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 400c54bdecfa44e08c7eca046bcd2ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:15.973094 6639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 400c54bdecfa44e08c7eca046bcd2ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:15.973284 6771 tablet_service.cc:1467] Processing CreateTablet for tablet efe57c7f306b4b0ab934388f5a372a74 (DEFAULT_TABLE table=rebalance_test_table_15 [id=65cfc473fc21464c991e7a62f281b04e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:15.973456 6640 tablet_service.cc:1467] Processing CreateTablet for tablet efe57c7f306b4b0ab934388f5a372a74 (DEFAULT_TABLE table=rebalance_test_table_15 [id=65cfc473fc21464c991e7a62f281b04e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:15.973520 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efe57c7f306b4b0ab934388f5a372a74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:15.973652 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efe57c7f306b4b0ab934388f5a372a74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:15.975361 8549 tablet_bootstrap.cc:492] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:15.975867 6769 tablet_service.cc:1467] Processing CreateTablet for tablet 2f3ac8b5d69745d788bb21a316e0ef21 (DEFAULT_TABLE table=rebalance_test_table_15 [id=65cfc473fc21464c991e7a62f281b04e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:15.976084 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f3ac8b5d69745d788bb21a316e0ef21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:15.976229 8549 tablet_bootstrap.cc:654] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:15.976342 8550 tablet_bootstrap.cc:492] T 400c54bdecfa44e08c7eca046bcd2ffc P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:15.977221 8550 tablet_bootstrap.cc:654] T 400c54bdecfa44e08c7eca046bcd2ffc P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:15.977165 6509 tablet_service.cc:1467] Processing CreateTablet for tablet 400c54bdecfa44e08c7eca046bcd2ffc (DEFAULT_TABLE table=rebalance_test_table_15 [id=65cfc473fc21464c991e7a62f281b04e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:15.977423 6509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 400c54bdecfa44e08c7eca046bcd2ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:15.978444 6901 tablet_service.cc:1467] Processing CreateTablet for tablet 2f3ac8b5d69745d788bb21a316e0ef21 (DEFAULT_TABLE table=rebalance_test_table_15 [id=65cfc473fc21464c991e7a62f281b04e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:15.978547 6900 tablet_service.cc:1467] Processing CreateTablet for tablet efe57c7f306b4b0ab934388f5a372a74 (DEFAULT_TABLE table=rebalance_test_table_15 [id=65cfc473fc21464c991e7a62f281b04e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:15.978693 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f3ac8b5d69745d788bb21a316e0ef21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:15.978864 6900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efe57c7f306b4b0ab934388f5a372a74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:15.979441 8549 tablet_bootstrap.cc:492] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:15.979597 8549 ts_tablet_manager.cc:1367] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:15.979960 6509 tablet_service.cc:1467] Processing CreateTablet for tablet 2f3ac8b5d69745d788bb21a316e0ef21 (DEFAULT_TABLE table=rebalance_test_table_15 [id=65cfc473fc21464c991e7a62f281b04e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:15.980099 8549 raft_consensus.cc:348] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:15.980173 6509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f3ac8b5d69745d788bb21a316e0ef21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:15.980201 8549 raft_consensus.cc:374] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:15.980232 8549 raft_consensus.cc:729] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:15.980347 8549 consensus_queue.cc:260] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:15.980664 8549 ts_tablet_manager.cc:1398] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:15.981669 8550 tablet_bootstrap.cc:492] T 400c54bdecfa44e08c7eca046bcd2ffc P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:15.981741 8550 ts_tablet_manager.cc:1367] T 400c54bdecfa44e08c7eca046bcd2ffc P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:59:15.982494 8550 raft_consensus.cc:348] T 400c54bdecfa44e08c7eca046bcd2ffc P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:15.982601 8550 raft_consensus.cc:374] T 400c54bdecfa44e08c7eca046bcd2ffc P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:15.982635 8550 raft_consensus.cc:729] T 400c54bdecfa44e08c7eca046bcd2ffc P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:15.982744 8550 consensus_queue.cc:260] T 400c54bdecfa44e08c7eca046bcd2ffc P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:15.982844 8551 tablet_bootstrap.cc:492] T efe57c7f306b4b0ab934388f5a372a74 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:15.983006 8550 ts_tablet_manager.cc:1398] T 400c54bdecfa44e08c7eca046bcd2ffc P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:15.983172 8550 tablet_bootstrap.cc:492] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:15.983450 8554 tablet_bootstrap.cc:492] T 400c54bdecfa44e08c7eca046bcd2ffc P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:15.984184 8550 tablet_bootstrap.cc:654] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:15.984299 8549 tablet_bootstrap.cc:492] T efe57c7f306b4b0ab934388f5a372a74 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:15.984567 8551 tablet_bootstrap.cc:654] T efe57c7f306b4b0ab934388f5a372a74 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:15.985354 8550 tablet_bootstrap.cc:492] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:15.985379 8549 tablet_bootstrap.cc:654] T efe57c7f306b4b0ab934388f5a372a74 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:15.985412 8554 tablet_bootstrap.cc:654] T 400c54bdecfa44e08c7eca046bcd2ffc P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:15.985430 8550 ts_tablet_manager.cc:1367] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:15.986058 8550 raft_consensus.cc:348] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:15.986150 8550 raft_consensus.cc:374] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:15.986178 8550 raft_consensus.cc:729] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:15.986330 8551 tablet_bootstrap.cc:492] T efe57c7f306b4b0ab934388f5a372a74 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:15.986289 8550 consensus_queue.cc:260] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:15.986398 8551 ts_tablet_manager.cc:1367] T efe57c7f306b4b0ab934388f5a372a74 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:15.986557 8550 ts_tablet_manager.cc:1398] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:15.986826 8551 raft_consensus.cc:348] T efe57c7f306b4b0ab934388f5a372a74 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:15.986930 8551 raft_consensus.cc:374] T efe57c7f306b4b0ab934388f5a372a74 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:15.986971 8551 raft_consensus.cc:729] T efe57c7f306b4b0ab934388f5a372a74 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:15.987074 8551 consensus_queue.cc:260] T efe57c7f306b4b0ab934388f5a372a74 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:15.987377 8551 ts_tablet_manager.cc:1398] T efe57c7f306b4b0ab934388f5a372a74 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:15.987454 8549 tablet_bootstrap.cc:492] T efe57c7f306b4b0ab934388f5a372a74 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:15.987525 8549 ts_tablet_manager.cc:1367] T efe57c7f306b4b0ab934388f5a372a74 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:15.987536 8551 tablet_bootstrap.cc:492] T 2f3ac8b5d69745d788bb21a316e0ef21 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:15.987793 8554 tablet_bootstrap.cc:492] T 400c54bdecfa44e08c7eca046bcd2ffc P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:15.987881 8554 ts_tablet_manager.cc:1367] T 400c54bdecfa44e08c7eca046bcd2ffc P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:59:15.987958 8549 raft_consensus.cc:348] T efe57c7f306b4b0ab934388f5a372a74 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:15.988062 8549 raft_consensus.cc:374] T efe57c7f306b4b0ab934388f5a372a74 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:15.988093 8549 raft_consensus.cc:729] T efe57c7f306b4b0ab934388f5a372a74 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:15.988195 8549 consensus_queue.cc:260] T efe57c7f306b4b0ab934388f5a372a74 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:15.988329 8554 raft_consensus.cc:348] T 400c54bdecfa44e08c7eca046bcd2ffc P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:15.988420 8554 raft_consensus.cc:374] T 400c54bdecfa44e08c7eca046bcd2ffc P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:15.988461 8549 ts_tablet_manager.cc:1398] T efe57c7f306b4b0ab934388f5a372a74 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:15.988451 8554 raft_consensus.cc:729] T 400c54bdecfa44e08c7eca046bcd2ffc P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:15.988584 8551 tablet_bootstrap.cc:654] T 2f3ac8b5d69745d788bb21a316e0ef21 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:15.988624 8549 tablet_bootstrap.cc:492] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:15.988593 8554 consensus_queue.cc:260] T 400c54bdecfa44e08c7eca046bcd2ffc P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:15.988885 8554 ts_tablet_manager.cc:1398] T 400c54bdecfa44e08c7eca046bcd2ffc P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:15.989046 8554 tablet_bootstrap.cc:492] T 2f3ac8b5d69745d788bb21a316e0ef21 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:15.989467 8549 tablet_bootstrap.cc:654] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:15.990053 8554 tablet_bootstrap.cc:654] T 2f3ac8b5d69745d788bb21a316e0ef21 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:15.990278 8551 tablet_bootstrap.cc:492] T 2f3ac8b5d69745d788bb21a316e0ef21 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:15.990352 8551 ts_tablet_manager.cc:1367] T 2f3ac8b5d69745d788bb21a316e0ef21 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:15.990759 8551 raft_consensus.cc:348] T 2f3ac8b5d69745d788bb21a316e0ef21 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:15.990831 8551 raft_consensus.cc:374] T 2f3ac8b5d69745d788bb21a316e0ef21 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:15.990851 8551 raft_consensus.cc:729] T 2f3ac8b5d69745d788bb21a316e0ef21 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:15.990940 8551 consensus_queue.cc:260] T 2f3ac8b5d69745d788bb21a316e0ef21 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:15.991189 8551 ts_tablet_manager.cc:1398] T 2f3ac8b5d69745d788bb21a316e0ef21 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:15.992787 8554 tablet_bootstrap.cc:492] T 2f3ac8b5d69745d788bb21a316e0ef21 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:15.992904 8554 ts_tablet_manager.cc:1367] T 2f3ac8b5d69745d788bb21a316e0ef21 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:15.993286 8554 raft_consensus.cc:348] T 2f3ac8b5d69745d788bb21a316e0ef21 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:15.993434 8554 raft_consensus.cc:374] T 2f3ac8b5d69745d788bb21a316e0ef21 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:15.993485 8554 raft_consensus.cc:729] T 2f3ac8b5d69745d788bb21a316e0ef21 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:15.993639 8554 consensus_queue.cc:260] T 2f3ac8b5d69745d788bb21a316e0ef21 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:15.993921 8554 ts_tablet_manager.cc:1398] T 2f3ac8b5d69745d788bb21a316e0ef21 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:15.995586 8549 tablet_bootstrap.cc:492] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:15.995725 8549 ts_tablet_manager.cc:1367] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 13:59:15.996201 8549 raft_consensus.cc:348] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:15.996296 8549 raft_consensus.cc:374] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:15.996331 8549 raft_consensus.cc:729] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:15.996444 8549 consensus_queue.cc:260] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:15.996699 8549 ts_tablet_manager.cc:1398] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:16.039387 8454 raft_consensus.cc:482] T 2f3ac8b5d69745d788bb21a316e0ef21 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:16.039525 8454 raft_consensus.cc:504] T 2f3ac8b5d69745d788bb21a316e0ef21 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:16.039916 8454 leader_election.cc:288] T 2f3ac8b5d69745d788bb21a316e0ef21 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:16.040377 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3ac8b5d69745d788bb21a316e0ef21" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:16.040414 6529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3ac8b5d69745d788bb21a316e0ef21" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:16.040539 6921 raft_consensus.cc:2454] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:16.040544 6529 raft_consensus.cc:2454] T 2f3ac8b5d69745d788bb21a316e0ef21 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:16.040815 6726 leader_election.cc:302] T 2f3ac8b5d69745d788bb21a316e0ef21 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:16.041003 8454 raft_consensus.cc:2781] T 2f3ac8b5d69745d788bb21a316e0ef21 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:16.041072 8454 raft_consensus.cc:482] T 2f3ac8b5d69745d788bb21a316e0ef21 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:16.041113 8454 raft_consensus.cc:3037] T 2f3ac8b5d69745d788bb21a316e0ef21 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:16.041934 8454 raft_consensus.cc:504] T 2f3ac8b5d69745d788bb21a316e0ef21 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:16.042241 8454 leader_election.cc:288] T 2f3ac8b5d69745d788bb21a316e0ef21 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:16.042605 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3ac8b5d69745d788bb21a316e0ef21" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:16.042676 6529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3ac8b5d69745d788bb21a316e0ef21" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:16.042709 6921 raft_consensus.cc:3037] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:16.042785 6529 raft_consensus.cc:3037] T 2f3ac8b5d69745d788bb21a316e0ef21 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:16.043555 6921 raft_consensus.cc:2454] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:16.043749 6529 raft_consensus.cc:2454] T 2f3ac8b5d69745d788bb21a316e0ef21 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:16.043853 6726 leader_election.cc:302] T 2f3ac8b5d69745d788bb21a316e0ef21 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:16.044081 8454 raft_consensus.cc:2781] T 2f3ac8b5d69745d788bb21a316e0ef21 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:16.044160 8454 raft_consensus.cc:686] T 2f3ac8b5d69745d788bb21a316e0ef21 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:16.044303 8454 consensus_queue.cc:237] T 2f3ac8b5d69745d788bb21a316e0ef21 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:16.046260 6379 catalog_manager.cc:5484] T 2f3ac8b5d69745d788bb21a316e0ef21 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:16.096738 8528 raft_consensus.cc:482] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:16.096858 8528 raft_consensus.cc:504] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:16.097242 8528 leader_election.cc:288] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:16.097554 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe57c7f306b4b0ab934388f5a372a74" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:16.097573 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe57c7f306b4b0ab934388f5a372a74" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:16.097700 6791 raft_consensus.cc:2454] T efe57c7f306b4b0ab934388f5a372a74 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:16.097700 6921 raft_consensus.cc:2454] T efe57c7f306b4b0ab934388f5a372a74 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:16.097944 6594 leader_election.cc:302] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:16.098124 8528 raft_consensus.cc:2781] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:16.098191 8528 raft_consensus.cc:482] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:16.098217 8528 raft_consensus.cc:3037] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:16.099203 8528 raft_consensus.cc:504] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:16.099568 8528 leader_election.cc:288] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:16.099891 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe57c7f306b4b0ab934388f5a372a74" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:16.100008 6791 raft_consensus.cc:3037] T efe57c7f306b4b0ab934388f5a372a74 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:16.100996 6791 raft_consensus.cc:2454] T efe57c7f306b4b0ab934388f5a372a74 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:16.101095 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe57c7f306b4b0ab934388f5a372a74" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:16.101222 6921 raft_consensus.cc:3037] T efe57c7f306b4b0ab934388f5a372a74 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:16.101909 6921 raft_consensus.cc:2454] T efe57c7f306b4b0ab934388f5a372a74 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:16.102202 6594 leader_election.cc:302] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:16.102357 8528 raft_consensus.cc:2781] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:16.102456 8528 raft_consensus.cc:686] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 13:59:16.102615 8528 consensus_queue.cc:237] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:16.104413 6379 catalog_manager.cc:5484] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:16.138796 8506 raft_consensus.cc:482] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:16.138916 8506 raft_consensus.cc:504] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:16.139253 8506 leader_election.cc:288] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:16.139628 6529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400c54bdecfa44e08c7eca046bcd2ffc" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:16.139649 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400c54bdecfa44e08c7eca046bcd2ffc" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:16.139780 6529 raft_consensus.cc:2454] T 400c54bdecfa44e08c7eca046bcd2ffc P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:16.139801 6660 raft_consensus.cc:2454] T 400c54bdecfa44e08c7eca046bcd2ffc P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:16.140051 6857 leader_election.cc:302] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:16.140220 8506 raft_consensus.cc:2781] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:16.140276 8506 raft_consensus.cc:482] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:16.140311 8506 raft_consensus.cc:3037] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:16.141122 8506 raft_consensus.cc:504] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:16.141431 8506 leader_election.cc:288] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:16.141708 6529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400c54bdecfa44e08c7eca046bcd2ffc" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:16.141731 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400c54bdecfa44e08c7eca046bcd2ffc" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:16.141825 6660 raft_consensus.cc:3037] T 400c54bdecfa44e08c7eca046bcd2ffc P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:16.141844 6529 raft_consensus.cc:3037] T 400c54bdecfa44e08c7eca046bcd2ffc P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:16.142711 6660 raft_consensus.cc:2454] T 400c54bdecfa44e08c7eca046bcd2ffc P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:16.142711 6529 raft_consensus.cc:2454] T 400c54bdecfa44e08c7eca046bcd2ffc P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:16.142979 6858 leader_election.cc:302] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:16.143170 8506 raft_consensus.cc:2781] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:16.143268 8506 raft_consensus.cc:686] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:16.143436 8506 consensus_queue.cc:237] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:16.145411 6379 catalog_manager.cc:5484] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:16.278299 8565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:16.278530 8565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:16.281458 8565 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_15'
I20240503 13:59:16.358031 6921 raft_consensus.cc:1261] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:16.358139 6529 raft_consensus.cc:1261] T 2f3ac8b5d69745d788bb21a316e0ef21 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:16.358716 8454 consensus_queue.cc:1035] T 2f3ac8b5d69745d788bb21a316e0ef21 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:16.360229 8454 consensus_queue.cc:1035] T 2f3ac8b5d69745d788bb21a316e0ef21 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:16.372166 6660 raft_consensus.cc:1261] T 400c54bdecfa44e08c7eca046bcd2ffc P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:16.372697 6922 raft_consensus.cc:1261] T efe57c7f306b4b0ab934388f5a372a74 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:16.373190 8506 consensus_queue.cc:1035] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:16.373564 6528 raft_consensus.cc:1261] T 400c54bdecfa44e08c7eca046bcd2ffc P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:16.374897 8506 consensus_queue.cc:1035] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:16.375811 6790 raft_consensus.cc:1261] T efe57c7f306b4b0ab934388f5a372a74 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:16.376690 8528 consensus_queue.cc:1035] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:16.378834 8333 consensus_queue.cc:1035] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 110.691 ms
time per row: 0.0553455 ms
I20240503 13:59:16.433490 6380 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:16.438742 6640 tablet_service.cc:1467] Processing CreateTablet for tablet e5588e9e339a498a984bee413013bdce (DEFAULT_TABLE table=rebalance_test_table_16 [id=c7a03df129af49a4a7f1052b1e4a915c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:16.438882 6901 tablet_service.cc:1467] Processing CreateTablet for tablet e5588e9e339a498a984bee413013bdce (DEFAULT_TABLE table=rebalance_test_table_16 [id=c7a03df129af49a4a7f1052b1e4a915c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:16.439075 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5588e9e339a498a984bee413013bdce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:16.439208 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5588e9e339a498a984bee413013bdce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:16.439409 6769 tablet_service.cc:1467] Processing CreateTablet for tablet b56aa48ef99c4cf89f4147c4f863005b (DEFAULT_TABLE table=rebalance_test_table_16 [id=c7a03df129af49a4a7f1052b1e4a915c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:16.439656 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b56aa48ef99c4cf89f4147c4f863005b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:16.440771 8550 tablet_bootstrap.cc:492] T e5588e9e339a498a984bee413013bdce P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:16.441006 6509 tablet_service.cc:1467] Processing CreateTablet for tablet e5588e9e339a498a984bee413013bdce (DEFAULT_TABLE table=rebalance_test_table_16 [id=c7a03df129af49a4a7f1052b1e4a915c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:16.441270 6509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5588e9e339a498a984bee413013bdce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:16.441356 8549 tablet_bootstrap.cc:492] T e5588e9e339a498a984bee413013bdce P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:16.441641 8550 tablet_bootstrap.cc:654] T e5588e9e339a498a984bee413013bdce P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:16.441717 8551 tablet_bootstrap.cc:492] T b56aa48ef99c4cf89f4147c4f863005b P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:16.442179 8549 tablet_bootstrap.cc:654] T e5588e9e339a498a984bee413013bdce P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:16.442528 8551 tablet_bootstrap.cc:654] T b56aa48ef99c4cf89f4147c4f863005b P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:16.443256 8549 tablet_bootstrap.cc:492] T e5588e9e339a498a984bee413013bdce P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:16.443370 8549 ts_tablet_manager.cc:1367] T e5588e9e339a498a984bee413013bdce P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 13:59:16.443511 6901 tablet_service.cc:1467] Processing CreateTablet for tablet 1c324cc92ff744a9a9283f64b471e347 (DEFAULT_TABLE table=rebalance_test_table_16 [id=c7a03df129af49a4a7f1052b1e4a915c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:16.443735 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c324cc92ff744a9a9283f64b471e347. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:16.443847 8554 tablet_bootstrap.cc:492] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:16.443925 6769 tablet_service.cc:1467] Processing CreateTablet for tablet 1c324cc92ff744a9a9283f64b471e347 (DEFAULT_TABLE table=rebalance_test_table_16 [id=c7a03df129af49a4a7f1052b1e4a915c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:16.443964 8549 raft_consensus.cc:348] T e5588e9e339a498a984bee413013bdce P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:16.444111 8549 raft_consensus.cc:374] T e5588e9e339a498a984bee413013bdce P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:16.444144 8549 raft_consensus.cc:729] T e5588e9e339a498a984bee413013bdce P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:16.444169 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c324cc92ff744a9a9283f64b471e347. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:16.444262 8549 consensus_queue.cc:260] T e5588e9e339a498a984bee413013bdce P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:16.444531 8549 ts_tablet_manager.cc:1398] T e5588e9e339a498a984bee413013bdce P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:16.444727 8554 tablet_bootstrap.cc:654] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:16.445477 6507 tablet_service.cc:1467] Processing CreateTablet for tablet 1c324cc92ff744a9a9283f64b471e347 (DEFAULT_TABLE table=rebalance_test_table_16 [id=c7a03df129af49a4a7f1052b1e4a915c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:16.445695 6507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c324cc92ff744a9a9283f64b471e347. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:16.448042 8551 tablet_bootstrap.cc:492] T b56aa48ef99c4cf89f4147c4f863005b P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:16.448117 8551 ts_tablet_manager.cc:1367] T b56aa48ef99c4cf89f4147c4f863005b P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 13:59:16.448559 8551 raft_consensus.cc:348] T b56aa48ef99c4cf89f4147c4f863005b P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:16.448678 8551 raft_consensus.cc:374] T b56aa48ef99c4cf89f4147c4f863005b P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:16.448710 8551 raft_consensus.cc:729] T b56aa48ef99c4cf89f4147c4f863005b P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:16.448824 8549 tablet_bootstrap.cc:492] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:16.448822 8551 consensus_queue.cc:260] T b56aa48ef99c4cf89f4147c4f863005b P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:16.449079 8551 ts_tablet_manager.cc:1398] T b56aa48ef99c4cf89f4147c4f863005b P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:16.449223 8551 tablet_bootstrap.cc:492] T 1c324cc92ff744a9a9283f64b471e347 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:16.449632 6640 tablet_service.cc:1467] Processing CreateTablet for tablet b56aa48ef99c4cf89f4147c4f863005b (DEFAULT_TABLE table=rebalance_test_table_16 [id=c7a03df129af49a4a7f1052b1e4a915c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:16.449872 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b56aa48ef99c4cf89f4147c4f863005b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:16.449853 6509 tablet_service.cc:1467] Processing CreateTablet for tablet b56aa48ef99c4cf89f4147c4f863005b (DEFAULT_TABLE table=rebalance_test_table_16 [id=c7a03df129af49a4a7f1052b1e4a915c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:16.449965 8551 tablet_bootstrap.cc:654] T 1c324cc92ff744a9a9283f64b471e347 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:16.450074 6509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b56aa48ef99c4cf89f4147c4f863005b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:16.451658 8551 tablet_bootstrap.cc:492] T 1c324cc92ff744a9a9283f64b471e347 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:16.451740 8551 ts_tablet_manager.cc:1367] T 1c324cc92ff744a9a9283f64b471e347 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:16.452288 8551 raft_consensus.cc:348] T 1c324cc92ff744a9a9283f64b471e347 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:16.452375 8551 raft_consensus.cc:374] T 1c324cc92ff744a9a9283f64b471e347 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:16.452411 8551 raft_consensus.cc:729] T 1c324cc92ff744a9a9283f64b471e347 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:16.452514 8551 consensus_queue.cc:260] T 1c324cc92ff744a9a9283f64b471e347 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:16.452780 8551 ts_tablet_manager.cc:1398] T 1c324cc92ff744a9a9283f64b471e347 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:16.453682 8549 tablet_bootstrap.cc:654] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:16.454831 8550 tablet_bootstrap.cc:492] T e5588e9e339a498a984bee413013bdce P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:16.454913 8550 ts_tablet_manager.cc:1367] T e5588e9e339a498a984bee413013bdce P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 13:59:16.455329 8550 raft_consensus.cc:348] T e5588e9e339a498a984bee413013bdce P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:16.455443 8550 raft_consensus.cc:374] T e5588e9e339a498a984bee413013bdce P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:16.455472 8550 raft_consensus.cc:729] T e5588e9e339a498a984bee413013bdce P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:16.455569 8550 consensus_queue.cc:260] T e5588e9e339a498a984bee413013bdce P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:16.455796 8550 ts_tablet_manager.cc:1398] T e5588e9e339a498a984bee413013bdce P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:16.455804 8549 tablet_bootstrap.cc:492] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:16.455873 8549 ts_tablet_manager.cc:1367] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 13:59:16.455929 8550 tablet_bootstrap.cc:492] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:16.456297 8549 raft_consensus.cc:348] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:16.456372 8549 raft_consensus.cc:374] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:16.456391 8549 raft_consensus.cc:729] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:16.456485 8549 consensus_queue.cc:260] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:16.456645 8550 tablet_bootstrap.cc:654] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:16.456759 8549 ts_tablet_manager.cc:1398] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:16.458846 8554 tablet_bootstrap.cc:492] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:16.458931 8554 ts_tablet_manager.cc:1367] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.000s
I20240503 13:59:16.459240 8554 raft_consensus.cc:348] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:16.459378 8554 raft_consensus.cc:374] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:16.459412 8554 raft_consensus.cc:729] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:16.459548 8554 consensus_queue.cc:260] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:16.459830 8554 ts_tablet_manager.cc:1398] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:16.460005 8554 tablet_bootstrap.cc:492] T 1c324cc92ff744a9a9283f64b471e347 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:16.460707 8554 tablet_bootstrap.cc:654] T 1c324cc92ff744a9a9283f64b471e347 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:16.462342 8554 tablet_bootstrap.cc:492] T 1c324cc92ff744a9a9283f64b471e347 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:16.462416 8554 ts_tablet_manager.cc:1367] T 1c324cc92ff744a9a9283f64b471e347 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:16.462826 8554 raft_consensus.cc:348] T 1c324cc92ff744a9a9283f64b471e347 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:16.462904 8554 raft_consensus.cc:374] T 1c324cc92ff744a9a9283f64b471e347 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:16.462925 8554 raft_consensus.cc:729] T 1c324cc92ff744a9a9283f64b471e347 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:16.463002 8554 consensus_queue.cc:260] T 1c324cc92ff744a9a9283f64b471e347 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:16.463212 8554 ts_tablet_manager.cc:1398] T 1c324cc92ff744a9a9283f64b471e347 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:16.463392 8554 tablet_bootstrap.cc:492] T b56aa48ef99c4cf89f4147c4f863005b P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:16.464568 8554 tablet_bootstrap.cc:654] T b56aa48ef99c4cf89f4147c4f863005b P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:16.466141 8554 tablet_bootstrap.cc:492] T b56aa48ef99c4cf89f4147c4f863005b P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:16.466214 8554 ts_tablet_manager.cc:1367] T b56aa48ef99c4cf89f4147c4f863005b P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:59:16.466559 8554 raft_consensus.cc:348] T b56aa48ef99c4cf89f4147c4f863005b P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:16.466634 8554 raft_consensus.cc:374] T b56aa48ef99c4cf89f4147c4f863005b P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:16.466667 8554 raft_consensus.cc:729] T b56aa48ef99c4cf89f4147c4f863005b P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:16.466818 8554 consensus_queue.cc:260] T b56aa48ef99c4cf89f4147c4f863005b P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:16.467037 8554 ts_tablet_manager.cc:1398] T b56aa48ef99c4cf89f4147c4f863005b P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:16.471891 8550 tablet_bootstrap.cc:492] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:16.471976 8550 ts_tablet_manager.cc:1367] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.000s
I20240503 13:59:16.472322 8550 raft_consensus.cc:348] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:16.472433 8550 raft_consensus.cc:374] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:16.472468 8550 raft_consensus.cc:729] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:16.472602 8550 consensus_queue.cc:260] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:16.472834 8550 ts_tablet_manager.cc:1398] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:16.553244 8423 raft_consensus.cc:482] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:16.553360 8423 raft_consensus.cc:504] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:16.553774 8423 leader_election.cc:288] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:16.554082 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b56aa48ef99c4cf89f4147c4f863005b" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:16.554160 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b56aa48ef99c4cf89f4147c4f863005b" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:16.554222 6791 raft_consensus.cc:2454] T b56aa48ef99c4cf89f4147c4f863005b P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:16.554298 6527 raft_consensus.cc:2454] T b56aa48ef99c4cf89f4147c4f863005b P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:16.554536 6594 leader_election.cc:302] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:16.554795 8423 raft_consensus.cc:2781] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:16.554864 8423 raft_consensus.cc:482] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:16.554904 8423 raft_consensus.cc:3037] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:16.555788 8423 raft_consensus.cc:504] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:16.555997 8600 raft_consensus.cc:482] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:16.556093 8600 raft_consensus.cc:504] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:16.556123 8423 leader_election.cc:288] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:16.556425 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b56aa48ef99c4cf89f4147c4f863005b" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:16.556525 6791 raft_consensus.cc:3037] T b56aa48ef99c4cf89f4147c4f863005b P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:16.556483 8600 leader_election.cc:288] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:16.556756 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c324cc92ff744a9a9283f64b471e347" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:16.556861 6790 raft_consensus.cc:2454] T 1c324cc92ff744a9a9283f64b471e347 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:16.557117 6856 leader_election.cc:302] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:16.557292 8600 raft_consensus.cc:2781] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:16.557350 8600 raft_consensus.cc:482] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:16.557404 8600 raft_consensus.cc:3037] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:16.557514 6791 raft_consensus.cc:2454] T b56aa48ef99c4cf89f4147c4f863005b P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:16.557754 6594 leader_election.cc:302] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:16.557862 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b56aa48ef99c4cf89f4147c4f863005b" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:16.557925 6529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c324cc92ff744a9a9283f64b471e347" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:16.557966 8423 raft_consensus.cc:2781] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:16.557986 6527 raft_consensus.cc:3037] T b56aa48ef99c4cf89f4147c4f863005b P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:16.558037 8423 raft_consensus.cc:686] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 13:59:16.558055 6529 raft_consensus.cc:2454] T 1c324cc92ff744a9a9283f64b471e347 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:16.558168 8423 consensus_queue.cc:237] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:16.558440 8600 raft_consensus.cc:504] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:16.558776 8600 leader_election.cc:288] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:16.558988 6527 raft_consensus.cc:2454] T b56aa48ef99c4cf89f4147c4f863005b P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:16.559084 6529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c324cc92ff744a9a9283f64b471e347" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:16.559175 6529 raft_consensus.cc:3037] T 1c324cc92ff744a9a9283f64b471e347 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:16.559933 6529 raft_consensus.cc:2454] T 1c324cc92ff744a9a9283f64b471e347 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:16.560115 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c324cc92ff744a9a9283f64b471e347" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:16.560220 6791 raft_consensus.cc:3037] T 1c324cc92ff744a9a9283f64b471e347 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:16.560868 6791 raft_consensus.cc:2454] T 1c324cc92ff744a9a9283f64b471e347 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:16.561149 6857 leader_election.cc:302] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:16.561318 8600 raft_consensus.cc:2781] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:16.561403 8600 raft_consensus.cc:686] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:16.561554 8600 consensus_queue.cc:237] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:16.561785 6383 catalog_manager.cc:5484] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:16.563725 6387 catalog_manager.cc:5484] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:16.703619 8182 raft_consensus.cc:482] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:16.703761 8182 raft_consensus.cc:504] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:16.704172 8182 leader_election.cc:288] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:16.704514 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5588e9e339a498a984bee413013bdce" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:16.704530 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5588e9e339a498a984bee413013bdce" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:16.704661 6660 raft_consensus.cc:2454] T e5588e9e339a498a984bee413013bdce P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:16.704659 6921 raft_consensus.cc:2454] T e5588e9e339a498a984bee413013bdce P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:16.704969 6464 leader_election.cc:302] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:16.705159 8182 raft_consensus.cc:2781] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:16.705219 8182 raft_consensus.cc:482] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:16.705260 8182 raft_consensus.cc:3037] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:16.706096 8182 raft_consensus.cc:504] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:16.706367 8182 leader_election.cc:288] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Requested vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:16.706714 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5588e9e339a498a984bee413013bdce" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:16.706734 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5588e9e339a498a984bee413013bdce" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:16.706837 6660 raft_consensus.cc:3037] T e5588e9e339a498a984bee413013bdce P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:16.706837 6921 raft_consensus.cc:3037] T e5588e9e339a498a984bee413013bdce P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:16.707803 6660 raft_consensus.cc:2454] T e5588e9e339a498a984bee413013bdce P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:16.707803 6921 raft_consensus.cc:2454] T e5588e9e339a498a984bee413013bdce P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:16.708081 6464 leader_election.cc:302] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:16.708246 8182 raft_consensus.cc:2781] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:16.708339 8182 raft_consensus.cc:686] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Becoming Leader. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Running, Role: LEADER
I20240503 13:59:16.708482 8182 consensus_queue.cc:237] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:16.710733 6387 catalog_manager.cc:5484] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:16.858414 8614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:16.858645 8614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:16.861706 8614 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_16'
I20240503 13:59:16.951444 6529 raft_consensus.cc:1261] T b56aa48ef99c4cf89f4147c4f863005b P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:16.951943 6791 raft_consensus.cc:1261] T b56aa48ef99c4cf89f4147c4f863005b P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:16.951938 8333 consensus_queue.cc:1035] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:16.952653 8333 consensus_queue.cc:1035] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:16.960222 6921 raft_consensus.cc:1261] T e5588e9e339a498a984bee413013bdce P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:16.960409 6527 raft_consensus.cc:1261] T 1c324cc92ff744a9a9283f64b471e347 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:16.961184 6660 raft_consensus.cc:1261] T e5588e9e339a498a984bee413013bdce P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:16.961552 8466 consensus_queue.cc:1035] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:16.961817 8466 consensus_queue.cc:1035] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:16.961958 6790 raft_consensus.cc:1261] T 1c324cc92ff744a9a9283f64b471e347 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:16.962705 8600 consensus_queue.cc:1035] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:16.965461 8506 consensus_queue.cc:1035] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 125.725 ms
time per row: 0.0628626 ms
I20240503 13:59:17.031756 6380 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:17.037279 6901 tablet_service.cc:1467] Processing CreateTablet for tablet 00434c9ea1654f7a9a9c31397a71998d (DEFAULT_TABLE table=rebalance_test_table_17 [id=f8de54141608449ab9dffc176985cb69]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:17.037616 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00434c9ea1654f7a9a9c31397a71998d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:17.037556 6769 tablet_service.cc:1467] Processing CreateTablet for tablet 00434c9ea1654f7a9a9c31397a71998d (DEFAULT_TABLE table=rebalance_test_table_17 [id=f8de54141608449ab9dffc176985cb69]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:17.037839 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00434c9ea1654f7a9a9c31397a71998d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:17.039547 8681 tablet_bootstrap.cc:492] T 00434c9ea1654f7a9a9c31397a71998d P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:17.039590 6771 tablet_service.cc:1467] Processing CreateTablet for tablet 7fde7d6bb07e4dbda5276b0dbbcff19b (DEFAULT_TABLE table=rebalance_test_table_17 [id=f8de54141608449ab9dffc176985cb69]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:17.039855 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fde7d6bb07e4dbda5276b0dbbcff19b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:17.040614 8681 tablet_bootstrap.cc:654] T 00434c9ea1654f7a9a9c31397a71998d P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:17.041816 8681 tablet_bootstrap.cc:492] T 00434c9ea1654f7a9a9c31397a71998d P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:17.041904 8681 ts_tablet_manager.cc:1367] T 00434c9ea1654f7a9a9c31397a71998d P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:17.041945 8683 tablet_bootstrap.cc:492] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:17.042327 8681 raft_consensus.cc:348] T 00434c9ea1654f7a9a9c31397a71998d P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:17.042470 8681 raft_consensus.cc:374] T 00434c9ea1654f7a9a9c31397a71998d P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:17.042740 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 00434c9ea1654f7a9a9c31397a71998d (DEFAULT_TABLE table=rebalance_test_table_17 [id=f8de54141608449ab9dffc176985cb69]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:17.042805 8681 raft_consensus.cc:729] T 00434c9ea1654f7a9a9c31397a71998d P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:17.042866 6638 tablet_service.cc:1467] Processing CreateTablet for tablet 39abc23bb3024e699e2f5bdb3bf5a7ea (DEFAULT_TABLE table=rebalance_test_table_17 [id=f8de54141608449ab9dffc176985cb69]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:17.042749 6639 tablet_service.cc:1467] Processing CreateTablet for tablet 7fde7d6bb07e4dbda5276b0dbbcff19b (DEFAULT_TABLE table=rebalance_test_table_17 [id=f8de54141608449ab9dffc176985cb69]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:17.042940 8681 consensus_queue.cc:260] T 00434c9ea1654f7a9a9c31397a71998d P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:17.043005 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00434c9ea1654f7a9a9c31397a71998d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:17.043118 6639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fde7d6bb07e4dbda5276b0dbbcff19b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:17.043249 8681 ts_tablet_manager.cc:1398] T 00434c9ea1654f7a9a9c31397a71998d P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:17.043330 6638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39abc23bb3024e699e2f5bdb3bf5a7ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:17.044996 8684 tablet_bootstrap.cc:492] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:17.045497 8683 tablet_bootstrap.cc:654] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:17.046308 8684 tablet_bootstrap.cc:654] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:17.047441 8684 tablet_bootstrap.cc:492] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:17.047441 6509 tablet_service.cc:1467] Processing CreateTablet for tablet 39abc23bb3024e699e2f5bdb3bf5a7ea (DEFAULT_TABLE table=rebalance_test_table_17 [id=f8de54141608449ab9dffc176985cb69]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:17.047529 8684 ts_tablet_manager.cc:1367] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:17.047705 6509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39abc23bb3024e699e2f5bdb3bf5a7ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:17.047686 6900 tablet_service.cc:1467] Processing CreateTablet for tablet 39abc23bb3024e699e2f5bdb3bf5a7ea (DEFAULT_TABLE table=rebalance_test_table_17 [id=f8de54141608449ab9dffc176985cb69]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:17.047904 6900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39abc23bb3024e699e2f5bdb3bf5a7ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:17.047981 8684 raft_consensus.cc:348] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:17.048070 8684 raft_consensus.cc:374] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:17.048097 8684 raft_consensus.cc:729] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:17.048218 8684 consensus_queue.cc:260] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:17.048476 8684 ts_tablet_manager.cc:1398] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:17.048780 6901 tablet_service.cc:1467] Processing CreateTablet for tablet 7fde7d6bb07e4dbda5276b0dbbcff19b (DEFAULT_TABLE table=rebalance_test_table_17 [id=f8de54141608449ab9dffc176985cb69]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:17.049005 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fde7d6bb07e4dbda5276b0dbbcff19b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:17.049629 8681 tablet_bootstrap.cc:492] T 39abc23bb3024e699e2f5bdb3bf5a7ea P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:17.050345 8687 tablet_bootstrap.cc:492] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:17.050536 8681 tablet_bootstrap.cc:654] T 39abc23bb3024e699e2f5bdb3bf5a7ea P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:17.051158 8687 tablet_bootstrap.cc:654] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:17.051189 8684 tablet_bootstrap.cc:492] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:17.052368 8684 tablet_bootstrap.cc:654] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:17.052551 8687 tablet_bootstrap.cc:492] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:17.052644 8687 ts_tablet_manager.cc:1367] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:17.053226 8687 raft_consensus.cc:348] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:17.053380 8684 tablet_bootstrap.cc:492] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:17.053412 8687 raft_consensus.cc:374] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:17.053450 8687 raft_consensus.cc:729] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:17.053463 8684 ts_tablet_manager.cc:1367] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:17.053732 8687 consensus_queue.cc:260] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:17.053892 8684 raft_consensus.cc:348] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:17.053978 8684 raft_consensus.cc:374] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:17.054008 8684 raft_consensus.cc:729] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:17.054035 8687 ts_tablet_manager.cc:1398] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:17.054116 8684 consensus_queue.cc:260] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:17.054348 8684 ts_tablet_manager.cc:1398] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:17.054461 8684 tablet_bootstrap.cc:492] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:17.055202 8684 tablet_bootstrap.cc:654] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:17.055240 8681 tablet_bootstrap.cc:492] T 39abc23bb3024e699e2f5bdb3bf5a7ea P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:17.055333 8681 ts_tablet_manager.cc:1367] T 39abc23bb3024e699e2f5bdb3bf5a7ea P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:59:17.055800 8681 raft_consensus.cc:348] T 39abc23bb3024e699e2f5bdb3bf5a7ea P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:17.055895 8681 raft_consensus.cc:374] T 39abc23bb3024e699e2f5bdb3bf5a7ea P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:17.055898 8683 tablet_bootstrap.cc:492] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:17.055923 8681 raft_consensus.cc:729] T 39abc23bb3024e699e2f5bdb3bf5a7ea P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:17.055986 8683 ts_tablet_manager.cc:1367] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 13:59:17.056027 8681 consensus_queue.cc:260] T 39abc23bb3024e699e2f5bdb3bf5a7ea P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:17.056274 8681 ts_tablet_manager.cc:1398] T 39abc23bb3024e699e2f5bdb3bf5a7ea P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:17.056432 8681 tablet_bootstrap.cc:492] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:17.056448 8683 raft_consensus.cc:348] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:17.056550 8683 raft_consensus.cc:374] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:17.056583 8683 raft_consensus.cc:729] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:17.056706 8683 consensus_queue.cc:260] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:17.057011 8683 ts_tablet_manager.cc:1398] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:17.057188 8683 tablet_bootstrap.cc:492] T 00434c9ea1654f7a9a9c31397a71998d P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:17.057308 8681 tablet_bootstrap.cc:654] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:17.057550 8684 tablet_bootstrap.cc:492] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:17.057626 8684 ts_tablet_manager.cc:1367] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:17.058013 8683 tablet_bootstrap.cc:654] T 00434c9ea1654f7a9a9c31397a71998d P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:17.058027 8684 raft_consensus.cc:348] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:17.058126 8684 raft_consensus.cc:374] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:17.058166 8684 raft_consensus.cc:729] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:17.058271 8684 consensus_queue.cc:260] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:17.058523 8684 ts_tablet_manager.cc:1398] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:17.058966 8681 tablet_bootstrap.cc:492] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:17.059036 8681 ts_tablet_manager.cc:1367] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:17.059496 8681 raft_consensus.cc:348] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:17.059609 8681 raft_consensus.cc:374] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:17.059659 8681 raft_consensus.cc:729] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:17.059765 8681 consensus_queue.cc:260] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:17.060014 8681 ts_tablet_manager.cc:1398] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:17.061594 8683 tablet_bootstrap.cc:492] T 00434c9ea1654f7a9a9c31397a71998d P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:17.061723 8683 ts_tablet_manager.cc:1367] T 00434c9ea1654f7a9a9c31397a71998d P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:17.062109 8683 raft_consensus.cc:348] T 00434c9ea1654f7a9a9c31397a71998d P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:17.062207 8683 raft_consensus.cc:374] T 00434c9ea1654f7a9a9c31397a71998d P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:17.062228 8683 raft_consensus.cc:729] T 00434c9ea1654f7a9a9c31397a71998d P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:17.062321 8683 consensus_queue.cc:260] T 00434c9ea1654f7a9a9c31397a71998d P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:17.062604 8683 ts_tablet_manager.cc:1398] T 00434c9ea1654f7a9a9c31397a71998d P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:17.091024 8656 raft_consensus.cc:482] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:17.091141 8656 raft_consensus.cc:504] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:17.091567 8656 leader_election.cc:288] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:17.091907 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39abc23bb3024e699e2f5bdb3bf5a7ea" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:17.091935 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39abc23bb3024e699e2f5bdb3bf5a7ea" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:17.092049 6659 raft_consensus.cc:2454] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:17.092046 6922 raft_consensus.cc:2454] T 39abc23bb3024e699e2f5bdb3bf5a7ea P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:17.092324 6465 leader_election.cc:302] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:17.092496 8656 raft_consensus.cc:2781] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:17.092561 8656 raft_consensus.cc:482] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:17.092619 8656 raft_consensus.cc:3037] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:17.093413 8656 raft_consensus.cc:504] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:17.093731 8656 leader_election.cc:288] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Requested vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:17.094077 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39abc23bb3024e699e2f5bdb3bf5a7ea" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:17.094077 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39abc23bb3024e699e2f5bdb3bf5a7ea" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:17.094220 6922 raft_consensus.cc:3037] T 39abc23bb3024e699e2f5bdb3bf5a7ea P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:17.094242 6659 raft_consensus.cc:3037] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:17.095203 6922 raft_consensus.cc:2454] T 39abc23bb3024e699e2f5bdb3bf5a7ea P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:17.095204 6659 raft_consensus.cc:2454] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:17.095486 6465 leader_election.cc:302] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:17.095698 8656 raft_consensus.cc:2781] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:17.095783 8656 raft_consensus.cc:686] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Becoming Leader. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Running, Role: LEADER
I20240503 13:59:17.095919 8656 consensus_queue.cc:237] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:17.098011 6383 catalog_manager.cc:5484] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:17.232127 8423 raft_consensus.cc:482] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:17.232259 8423 raft_consensus.cc:504] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:17.232717 8423 leader_election.cc:288] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:17.233098 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00434c9ea1654f7a9a9c31397a71998d" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:17.233100 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00434c9ea1654f7a9a9c31397a71998d" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:17.233232 6790 raft_consensus.cc:2454] T 00434c9ea1654f7a9a9c31397a71998d P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:17.233237 6922 raft_consensus.cc:2454] T 00434c9ea1654f7a9a9c31397a71998d P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:17.233541 6594 leader_election.cc:302] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:17.233742 8423 raft_consensus.cc:2781] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:17.233840 8423 raft_consensus.cc:482] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:17.233901 8423 raft_consensus.cc:3037] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:17.234692 8423 raft_consensus.cc:504] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:17.235019 8423 leader_election.cc:288] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:17.235365 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00434c9ea1654f7a9a9c31397a71998d" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:17.235440 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00434c9ea1654f7a9a9c31397a71998d" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:17.235482 6922 raft_consensus.cc:3037] T 00434c9ea1654f7a9a9c31397a71998d P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:17.235558 6790 raft_consensus.cc:3037] T 00434c9ea1654f7a9a9c31397a71998d P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:17.236248 6922 raft_consensus.cc:2454] T 00434c9ea1654f7a9a9c31397a71998d P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:17.236312 6790 raft_consensus.cc:2454] T 00434c9ea1654f7a9a9c31397a71998d P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:17.236548 6595 leader_election.cc:302] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:17.236716 8423 raft_consensus.cc:2781] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:17.236804 8423 raft_consensus.cc:686] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 13:59:17.236980 8423 consensus_queue.cc:237] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:17.239017 6383 catalog_manager.cc:5484] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:17.351873 8506 raft_consensus.cc:482] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:17.352000 8506 raft_consensus.cc:504] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:17.352416 8506 leader_election.cc:288] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:17.352820 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fde7d6bb07e4dbda5276b0dbbcff19b" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:17.352821 6789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fde7d6bb07e4dbda5276b0dbbcff19b" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:17.352941 6659 raft_consensus.cc:2454] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:17.352964 6789 raft_consensus.cc:2454] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:17.353235 6858 leader_election.cc:302] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:17.353457 8506 raft_consensus.cc:2781] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:17.353521 8506 raft_consensus.cc:482] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:17.353552 8506 raft_consensus.cc:3037] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:17.354328 8506 raft_consensus.cc:504] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:17.354660 8506 leader_election.cc:288] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:17.355006 6789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fde7d6bb07e4dbda5276b0dbbcff19b" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:17.355057 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fde7d6bb07e4dbda5276b0dbbcff19b" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:17.355129 6789 raft_consensus.cc:3037] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:17.355221 6659 raft_consensus.cc:3037] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:17.356134 6789 raft_consensus.cc:2454] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:17.356225 6659 raft_consensus.cc:2454] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:17.356526 6858 leader_election.cc:302] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:17.356703 8506 raft_consensus.cc:2781] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:17.356776 8506 raft_consensus.cc:686] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:17.356932 8506 consensus_queue.cc:237] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:17.359088 6383 catalog_manager.cc:5484] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:17.525403 8698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:17.525635 8698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:17.528879 8698 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 13:59:17.538498 8656 consensus_queue.cc:1035] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:17.542210 8656 consensus_queue.cc:1035] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_17'
I20240503 13:59:17.618093 6922 raft_consensus.cc:1261] T 00434c9ea1654f7a9a9c31397a71998d P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:17.618147 6789 raft_consensus.cc:1261] T 00434c9ea1654f7a9a9c31397a71998d P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:17.618795 8333 consensus_queue.cc:1035] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:17.621024 8423 consensus_queue.cc:1035] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:17.624156 6790 raft_consensus.cc:1261] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:17.625039 8506 consensus_queue.cc:1035] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:17.625792 6660 raft_consensus.cc:1261] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:17.626601 8506 consensus_queue.cc:1035] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:17.664546 8738 mvcc.cc:205] Tried to move back new op lower bound from 7023594527223128064 to 7023594526135328768. Current Snapshot: MvccSnapshot[applied={T|T < 7023594527223128064}]
INSERT report
rows total: 2000
time total: 126.337 ms
time per row: 0.0631683 ms
I20240503 13:59:17.700464 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:17.706528 6769 tablet_service.cc:1467] Processing CreateTablet for tablet b05fc3d7abb04aa086ec4caf7eabb6ab (DEFAULT_TABLE table=rebalance_test_table_18 [id=3057a1d5b4be45e49902b304f0a52e14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:17.706861 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b05fc3d7abb04aa086ec4caf7eabb6ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:17.706954 6771 tablet_service.cc:1467] Processing CreateTablet for tablet 0162cfee86ed4f3394e6829a405037fe (DEFAULT_TABLE table=rebalance_test_table_18 [id=3057a1d5b4be45e49902b304f0a52e14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:17.707296 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0162cfee86ed4f3394e6829a405037fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:17.707614 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 0d75271c2d2d4aa88f7e499898300782 (DEFAULT_TABLE table=rebalance_test_table_18 [id=3057a1d5b4be45e49902b304f0a52e14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:17.707862 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d75271c2d2d4aa88f7e499898300782. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:17.707998 6901 tablet_service.cc:1467] Processing CreateTablet for tablet 0162cfee86ed4f3394e6829a405037fe (DEFAULT_TABLE table=rebalance_test_table_18 [id=3057a1d5b4be45e49902b304f0a52e14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:17.708287 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0162cfee86ed4f3394e6829a405037fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:17.709331 6639 tablet_service.cc:1467] Processing CreateTablet for tablet b05fc3d7abb04aa086ec4caf7eabb6ab (DEFAULT_TABLE table=rebalance_test_table_18 [id=3057a1d5b4be45e49902b304f0a52e14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:17.709559 6639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b05fc3d7abb04aa086ec4caf7eabb6ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:17.709676 6771 tablet_service.cc:1467] Processing CreateTablet for tablet 0d75271c2d2d4aa88f7e499898300782 (DEFAULT_TABLE table=rebalance_test_table_18 [id=3057a1d5b4be45e49902b304f0a52e14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:17.709864 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d75271c2d2d4aa88f7e499898300782. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:17.710685 8764 tablet_bootstrap.cc:492] T 0162cfee86ed4f3394e6829a405037fe P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:17.711773 8764 tablet_bootstrap.cc:654] T 0162cfee86ed4f3394e6829a405037fe P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:17.711892 8762 tablet_bootstrap.cc:492] T b05fc3d7abb04aa086ec4caf7eabb6ab P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:17.711932 6509 tablet_service.cc:1467] Processing CreateTablet for tablet b05fc3d7abb04aa086ec4caf7eabb6ab (DEFAULT_TABLE table=rebalance_test_table_18 [id=3057a1d5b4be45e49902b304f0a52e14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:17.712209 6509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b05fc3d7abb04aa086ec4caf7eabb6ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:17.712735 8764 tablet_bootstrap.cc:492] T 0162cfee86ed4f3394e6829a405037fe P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:17.712795 8764 ts_tablet_manager.cc:1367] T 0162cfee86ed4f3394e6829a405037fe P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:17.712937 8762 tablet_bootstrap.cc:654] T b05fc3d7abb04aa086ec4caf7eabb6ab P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:17.713167 8764 raft_consensus.cc:348] T 0162cfee86ed4f3394e6829a405037fe P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:17.713258 8764 raft_consensus.cc:374] T 0162cfee86ed4f3394e6829a405037fe P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:17.713279 8764 raft_consensus.cc:729] T 0162cfee86ed4f3394e6829a405037fe P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:17.713383 8764 consensus_queue.cc:260] T 0162cfee86ed4f3394e6829a405037fe P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:17.713642 8764 ts_tablet_manager.cc:1398] T 0162cfee86ed4f3394e6829a405037fe P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:17.714468 8767 tablet_bootstrap.cc:492] T b05fc3d7abb04aa086ec4caf7eabb6ab P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:17.714686 6507 tablet_service.cc:1467] Processing CreateTablet for tablet 0162cfee86ed4f3394e6829a405037fe (DEFAULT_TABLE table=rebalance_test_table_18 [id=3057a1d5b4be45e49902b304f0a52e14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:17.714883 6507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0162cfee86ed4f3394e6829a405037fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:17.715327 8767 tablet_bootstrap.cc:654] T b05fc3d7abb04aa086ec4caf7eabb6ab P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:17.715603 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 0d75271c2d2d4aa88f7e499898300782 (DEFAULT_TABLE table=rebalance_test_table_18 [id=3057a1d5b4be45e49902b304f0a52e14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:17.715754 8762 tablet_bootstrap.cc:492] T b05fc3d7abb04aa086ec4caf7eabb6ab P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:17.715837 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d75271c2d2d4aa88f7e499898300782. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:17.715844 8762 ts_tablet_manager.cc:1367] T b05fc3d7abb04aa086ec4caf7eabb6ab P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:17.716301 8762 raft_consensus.cc:348] T b05fc3d7abb04aa086ec4caf7eabb6ab P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:17.716428 8762 raft_consensus.cc:374] T b05fc3d7abb04aa086ec4caf7eabb6ab P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:17.716459 8762 raft_consensus.cc:729] T b05fc3d7abb04aa086ec4caf7eabb6ab P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:17.716575 8762 consensus_queue.cc:260] T b05fc3d7abb04aa086ec4caf7eabb6ab P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:17.716885 8762 ts_tablet_manager.cc:1398] T b05fc3d7abb04aa086ec4caf7eabb6ab P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:17.717159 8762 tablet_bootstrap.cc:492] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:17.717348 8767 tablet_bootstrap.cc:492] T b05fc3d7abb04aa086ec4caf7eabb6ab P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:17.717419 8767 ts_tablet_manager.cc:1367] T b05fc3d7abb04aa086ec4caf7eabb6ab P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:59:17.718101 8762 tablet_bootstrap.cc:654] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:17.718643 8758 raft_consensus.cc:482] T 0162cfee86ed4f3394e6829a405037fe P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:17.718739 8758 raft_consensus.cc:504] T 0162cfee86ed4f3394e6829a405037fe P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:17.719130 8758 leader_election.cc:288] T 0162cfee86ed4f3394e6829a405037fe P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:17.719575 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0162cfee86ed4f3394e6829a405037fe" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:17.720125 8763 tablet_bootstrap.cc:492] T 0d75271c2d2d4aa88f7e499898300782 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:17.721091 8763 tablet_bootstrap.cc:654] T 0d75271c2d2d4aa88f7e499898300782 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
W20240503 13:59:17.721113 6856 leader_election.cc:341] T 0162cfee86ed4f3394e6829a405037fe P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:59:17.721668 6529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0162cfee86ed4f3394e6829a405037fe" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:17.721705 8767 raft_consensus.cc:348] T b05fc3d7abb04aa086ec4caf7eabb6ab P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:17.721781 8767 raft_consensus.cc:374] T b05fc3d7abb04aa086ec4caf7eabb6ab P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:17.721819 8767 raft_consensus.cc:729] T b05fc3d7abb04aa086ec4caf7eabb6ab P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
W20240503 13:59:17.722084 6857 leader_election.cc:341] T 0162cfee86ed4f3394e6829a405037fe P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:59:17.722165 6857 leader_election.cc:302] T 0162cfee86ed4f3394e6829a405037fe P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 4a85e492c8154e9881a9cc2a105d7e64, 5f46e19644fd4f4b825c86ba6af768da
I20240503 13:59:17.722290 8758 raft_consensus.cc:2726] T 0162cfee86ed4f3394e6829a405037fe P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 13:59:17.722416 8767 consensus_queue.cc:260] T b05fc3d7abb04aa086ec4caf7eabb6ab P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:17.722724 8767 ts_tablet_manager.cc:1398] T b05fc3d7abb04aa086ec4caf7eabb6ab P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240503 13:59:17.722952 8767 tablet_bootstrap.cc:492] T 0162cfee86ed4f3394e6829a405037fe P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:17.723151 8763 tablet_bootstrap.cc:492] T 0d75271c2d2d4aa88f7e499898300782 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:17.723474 8763 ts_tablet_manager.cc:1367] T 0d75271c2d2d4aa88f7e499898300782 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:17.723883 8767 tablet_bootstrap.cc:654] T 0162cfee86ed4f3394e6829a405037fe P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:17.724009 8763 raft_consensus.cc:348] T 0d75271c2d2d4aa88f7e499898300782 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:17.724112 8763 raft_consensus.cc:374] T 0d75271c2d2d4aa88f7e499898300782 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:17.724177 8763 raft_consensus.cc:729] T 0d75271c2d2d4aa88f7e499898300782 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:17.724316 8763 consensus_queue.cc:260] T 0d75271c2d2d4aa88f7e499898300782 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:17.725051 8767 tablet_bootstrap.cc:492] T 0162cfee86ed4f3394e6829a405037fe P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:17.725124 8767 ts_tablet_manager.cc:1367] T 0162cfee86ed4f3394e6829a405037fe P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:17.725311 8763 ts_tablet_manager.cc:1398] T 0d75271c2d2d4aa88f7e499898300782 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:17.725474 8763 tablet_bootstrap.cc:492] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:17.725526 8767 raft_consensus.cc:348] T 0162cfee86ed4f3394e6829a405037fe P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:17.725605 8767 raft_consensus.cc:374] T 0162cfee86ed4f3394e6829a405037fe P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:17.725633 8767 raft_consensus.cc:729] T 0162cfee86ed4f3394e6829a405037fe P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:17.725888 8767 consensus_queue.cc:260] T 0162cfee86ed4f3394e6829a405037fe P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:17.726146 8767 ts_tablet_manager.cc:1398] T 0162cfee86ed4f3394e6829a405037fe P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:17.726294 8767 tablet_bootstrap.cc:492] T 0d75271c2d2d4aa88f7e499898300782 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:17.726374 8763 tablet_bootstrap.cc:654] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:17.727305 8767 tablet_bootstrap.cc:654] T 0d75271c2d2d4aa88f7e499898300782 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:17.728291 8762 tablet_bootstrap.cc:492] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:17.728389 8767 tablet_bootstrap.cc:492] T 0d75271c2d2d4aa88f7e499898300782 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:17.728422 8762 ts_tablet_manager.cc:1367] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 13:59:17.728467 8767 ts_tablet_manager.cc:1367] T 0d75271c2d2d4aa88f7e499898300782 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:17.728905 8762 raft_consensus.cc:348] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:17.728973 8767 raft_consensus.cc:348] T 0d75271c2d2d4aa88f7e499898300782 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:17.729064 8767 raft_consensus.cc:374] T 0d75271c2d2d4aa88f7e499898300782 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:17.729074 8763 tablet_bootstrap.cc:492] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:17.729108 8767 raft_consensus.cc:729] T 0d75271c2d2d4aa88f7e499898300782 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:17.729136 8763 ts_tablet_manager.cc:1367] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 13:59:17.729147 8762 raft_consensus.cc:374] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:17.729245 8767 consensus_queue.cc:260] T 0d75271c2d2d4aa88f7e499898300782 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:17.729465 8762 raft_consensus.cc:729] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:17.729454 8763 raft_consensus.cc:348] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:17.729517 8767 ts_tablet_manager.cc:1398] T 0d75271c2d2d4aa88f7e499898300782 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:17.729604 8763 raft_consensus.cc:374] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:17.729673 8763 raft_consensus.cc:729] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:17.729714 8762 consensus_queue.cc:260] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:17.729851 8763 consensus_queue.cc:260] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:17.730103 8762 ts_tablet_manager.cc:1398] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:17.730290 8763 ts_tablet_manager.cc:1398] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:17.730835 8762 tablet_bootstrap.cc:492] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:17.731665 8762 tablet_bootstrap.cc:654] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:17.733383 8762 tablet_bootstrap.cc:492] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:17.733466 8762 ts_tablet_manager.cc:1367] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:17.733916 8762 raft_consensus.cc:348] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:17.734033 8762 raft_consensus.cc:374] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:17.734071 8762 raft_consensus.cc:729] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:17.734182 8762 consensus_queue.cc:260] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:17.734459 8762 ts_tablet_manager.cc:1398] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:17.739542 8394 raft_consensus.cc:482] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:17.739643 8394 raft_consensus.cc:504] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:17.739955 8394 leader_election.cc:288] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:17.740245 6529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d75271c2d2d4aa88f7e499898300782" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:17.740303 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d75271c2d2d4aa88f7e499898300782" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:17.740373 6529 raft_consensus.cc:2454] T 0d75271c2d2d4aa88f7e499898300782 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:17.740437 6660 raft_consensus.cc:2454] T 0d75271c2d2d4aa88f7e499898300782 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:17.740638 6726 leader_election.cc:302] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:17.740835 8394 raft_consensus.cc:2781] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:17.740904 8394 raft_consensus.cc:482] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:17.740942 8394 raft_consensus.cc:3037] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:17.741698 8394 raft_consensus.cc:504] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:17.742048 8394 leader_election.cc:288] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:17.742378 6529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d75271c2d2d4aa88f7e499898300782" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:17.742496 6529 raft_consensus.cc:3037] T 0d75271c2d2d4aa88f7e499898300782 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:17.742512 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d75271c2d2d4aa88f7e499898300782" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:17.742623 6659 raft_consensus.cc:3037] T 0d75271c2d2d4aa88f7e499898300782 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:17.743530 6529 raft_consensus.cc:2454] T 0d75271c2d2d4aa88f7e499898300782 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:17.743602 6659 raft_consensus.cc:2454] T 0d75271c2d2d4aa88f7e499898300782 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:17.743861 6726 leader_election.cc:302] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:17.744019 8394 raft_consensus.cc:2781] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:17.744092 8394 raft_consensus.cc:686] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:17.744238 8394 consensus_queue.cc:237] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:17.746281 6383 catalog_manager.cc:5484] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:17.786631 8394 raft_consensus.cc:482] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:17.786757 8394 raft_consensus.cc:504] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:17.786823 8423 raft_consensus.cc:482] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:17.786909 8423 raft_consensus.cc:504] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:17.787175 8394 leader_election.cc:288] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:17.787313 8423 leader_election.cc:288] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:17.787511 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0162cfee86ed4f3394e6829a405037fe" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:17.787650 6921 raft_consensus.cc:2454] T 0162cfee86ed4f3394e6829a405037fe P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:17.787643 6529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b05fc3d7abb04aa086ec4caf7eabb6ab" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:17.787707 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0162cfee86ed4f3394e6829a405037fe" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:17.787796 6529 raft_consensus.cc:2454] T b05fc3d7abb04aa086ec4caf7eabb6ab P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:17.787783 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b05fc3d7abb04aa086ec4caf7eabb6ab" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:17.787833 6527 raft_consensus.cc:2454] T 0162cfee86ed4f3394e6829a405037fe P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:17.787887 6791 raft_consensus.cc:2454] T b05fc3d7abb04aa086ec4caf7eabb6ab P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:17.788146 6595 leader_election.cc:302] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:17.788300 8423 raft_consensus.cc:2781] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:17.788324 6726 leader_election.cc:302] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:17.788352 8423 raft_consensus.cc:482] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:17.788389 8423 raft_consensus.cc:3037] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:17.788508 8696 raft_consensus.cc:2781] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:17.788564 8696 raft_consensus.cc:482] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:17.788615 8696 raft_consensus.cc:3037] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:17.789469 8423 raft_consensus.cc:504] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:17.789566 8696 raft_consensus.cc:504] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:17.789817 8423 leader_election.cc:288] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:17.789861 8696 leader_election.cc:288] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:17.790145 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0162cfee86ed4f3394e6829a405037fe" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:17.790155 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b05fc3d7abb04aa086ec4caf7eabb6ab" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:17.790262 6921 raft_consensus.cc:3037] T 0162cfee86ed4f3394e6829a405037fe P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:17.790290 6791 raft_consensus.cc:3037] T b05fc3d7abb04aa086ec4caf7eabb6ab P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:17.790345 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b05fc3d7abb04aa086ec4caf7eabb6ab" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:17.790370 6529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0162cfee86ed4f3394e6829a405037fe" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:17.790445 6528 raft_consensus.cc:3037] T b05fc3d7abb04aa086ec4caf7eabb6ab P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:17.790474 6529 raft_consensus.cc:3037] T 0162cfee86ed4f3394e6829a405037fe P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:17.791208 6921 raft_consensus.cc:2454] T 0162cfee86ed4f3394e6829a405037fe P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:17.791376 6529 raft_consensus.cc:2454] T 0162cfee86ed4f3394e6829a405037fe P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:17.791461 6528 raft_consensus.cc:2454] T b05fc3d7abb04aa086ec4caf7eabb6ab P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:17.791776 6595 leader_election.cc:302] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:17.791790 6726 leader_election.cc:302] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:17.791936 8696 raft_consensus.cc:2781] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:17.791986 8423 raft_consensus.cc:2781] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:17.792009 8696 raft_consensus.cc:686] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:17.792061 8423 raft_consensus.cc:686] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 13:59:17.792140 8696 consensus_queue.cc:237] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:17.792163 8423 consensus_queue.cc:237] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:17.791216 6791 raft_consensus.cc:2454] T b05fc3d7abb04aa086ec4caf7eabb6ab P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:17.794384 6382 catalog_manager.cc:5484] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:17.794872 6383 catalog_manager.cc:5484] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } }
W20240503 13:59:17.912844 8779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:17.913125 8779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:17.917104 8779 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_18'
I20240503 13:59:17.997390 6659 raft_consensus.cc:1261] T 0d75271c2d2d4aa88f7e499898300782 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:17.997838 8806 consensus_queue.cc:1035] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:17.998545 6529 raft_consensus.cc:1261] T 0162cfee86ed4f3394e6829a405037fe P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:17.998582 6791 raft_consensus.cc:1261] T b05fc3d7abb04aa086ec4caf7eabb6ab P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:17.998903 6527 raft_consensus.cc:1261] T 0d75271c2d2d4aa88f7e499898300782 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:17.998920 6921 raft_consensus.cc:1261] T 0162cfee86ed4f3394e6829a405037fe P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:17.998545 6528 raft_consensus.cc:1261] T b05fc3d7abb04aa086ec4caf7eabb6ab P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:17.999832 8806 consensus_queue.cc:1035] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:17.999867 8754 consensus_queue.cc:1035] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:18.005544 8394 consensus_queue.cc:1035] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:18.005815 8394 consensus_queue.cc:1035] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:18.007817 8423 consensus_queue.cc:1035] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:18.025622 8709 mvcc.cc:205] Tried to move back new op lower bound from 7023594528780042240 to 7023594527917666304. Current Snapshot: MvccSnapshot[applied={T|T < 7023594528746897408}]
I20240503 13:59:18.029963 8747 mvcc.cc:205] Tried to move back new op lower bound from 7023594528741916672 to 7023594527917797376. Current Snapshot: MvccSnapshot[applied={T|T < 7023594528741916672}]
I20240503 13:59:18.061058 8746 mvcc.cc:205] Tried to move back new op lower bound from 7023594528752455680 to 7023594527721684992. Current Snapshot: MvccSnapshot[applied={T|T < 7023594528752455680}]
INSERT report
rows total: 2000
time total: 119.166 ms
time per row: 0.059583 ms
I20240503 13:59:18.082895 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:18.088086 6901 tablet_service.cc:1467] Processing CreateTablet for tablet b6cb5119a6db403ba3cae9fa9b121211 (DEFAULT_TABLE table=rebalance_test_table_19 [id=aaf095359a4a40ecbae916f4245a8f5f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:18.088330 6639 tablet_service.cc:1467] Processing CreateTablet for tablet b6cb5119a6db403ba3cae9fa9b121211 (DEFAULT_TABLE table=rebalance_test_table_19 [id=aaf095359a4a40ecbae916f4245a8f5f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:18.088445 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6cb5119a6db403ba3cae9fa9b121211. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:18.088603 6639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6cb5119a6db403ba3cae9fa9b121211. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:18.088851 6508 tablet_service.cc:1467] Processing CreateTablet for tablet b6cb5119a6db403ba3cae9fa9b121211 (DEFAULT_TABLE table=rebalance_test_table_19 [id=aaf095359a4a40ecbae916f4245a8f5f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:18.088927 6507 tablet_service.cc:1467] Processing CreateTablet for tablet f9d875f4872e4f5da12a326381d523c0 (DEFAULT_TABLE table=rebalance_test_table_19 [id=aaf095359a4a40ecbae916f4245a8f5f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:18.089092 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6cb5119a6db403ba3cae9fa9b121211. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:18.089251 6507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9d875f4872e4f5da12a326381d523c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:18.089746 6771 tablet_service.cc:1467] Processing CreateTablet for tablet f9d875f4872e4f5da12a326381d523c0 (DEFAULT_TABLE table=rebalance_test_table_19 [id=aaf095359a4a40ecbae916f4245a8f5f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:18.089993 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9d875f4872e4f5da12a326381d523c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:18.090874 6900 tablet_service.cc:1467] Processing CreateTablet for tablet f9d875f4872e4f5da12a326381d523c0 (DEFAULT_TABLE table=rebalance_test_table_19 [id=aaf095359a4a40ecbae916f4245a8f5f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:18.091116 6900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9d875f4872e4f5da12a326381d523c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:18.091249 8767 tablet_bootstrap.cc:492] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:18.091552 6902 tablet_service.cc:1467] Processing CreateTablet for tablet 3612cd35d8fa4300a1132fe43a63e6c1 (DEFAULT_TABLE table=rebalance_test_table_19 [id=aaf095359a4a40ecbae916f4245a8f5f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:18.091763 6902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3612cd35d8fa4300a1132fe43a63e6c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:18.092126 8767 tablet_bootstrap.cc:654] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:18.092303 6769 tablet_service.cc:1467] Processing CreateTablet for tablet 3612cd35d8fa4300a1132fe43a63e6c1 (DEFAULT_TABLE table=rebalance_test_table_19 [id=aaf095359a4a40ecbae916f4245a8f5f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:18.092502 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3612cd35d8fa4300a1132fe43a63e6c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:18.093708 8762 tablet_bootstrap.cc:492] T f9d875f4872e4f5da12a326381d523c0 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:18.094324 8764 tablet_bootstrap.cc:492] T b6cb5119a6db403ba3cae9fa9b121211 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:18.094440 8762 tablet_bootstrap.cc:654] T f9d875f4872e4f5da12a326381d523c0 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:18.094960 8764 tablet_bootstrap.cc:654] T b6cb5119a6db403ba3cae9fa9b121211 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:18.095456 8762 tablet_bootstrap.cc:492] T f9d875f4872e4f5da12a326381d523c0 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:18.095530 8762 ts_tablet_manager.cc:1367] T f9d875f4872e4f5da12a326381d523c0 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:18.095563 8763 tablet_bootstrap.cc:492] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:18.095520 6639 tablet_service.cc:1467] Processing CreateTablet for tablet 3612cd35d8fa4300a1132fe43a63e6c1 (DEFAULT_TABLE table=rebalance_test_table_19 [id=aaf095359a4a40ecbae916f4245a8f5f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:18.095789 6639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3612cd35d8fa4300a1132fe43a63e6c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:18.095935 8762 raft_consensus.cc:348] T f9d875f4872e4f5da12a326381d523c0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:18.096038 8762 raft_consensus.cc:374] T f9d875f4872e4f5da12a326381d523c0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:18.096065 8762 raft_consensus.cc:729] T f9d875f4872e4f5da12a326381d523c0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:18.096171 8762 consensus_queue.cc:260] T f9d875f4872e4f5da12a326381d523c0 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:18.096378 8763 tablet_bootstrap.cc:654] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:18.096443 8762 ts_tablet_manager.cc:1398] T f9d875f4872e4f5da12a326381d523c0 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:18.096580 8762 tablet_bootstrap.cc:492] T 3612cd35d8fa4300a1132fe43a63e6c1 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:18.097409 8762 tablet_bootstrap.cc:654] T 3612cd35d8fa4300a1132fe43a63e6c1 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:18.099699 8764 tablet_bootstrap.cc:492] T b6cb5119a6db403ba3cae9fa9b121211 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:18.099745 8762 tablet_bootstrap.cc:492] T 3612cd35d8fa4300a1132fe43a63e6c1 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:18.099789 8764 ts_tablet_manager.cc:1367] T b6cb5119a6db403ba3cae9fa9b121211 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 13:59:18.099812 8762 ts_tablet_manager.cc:1367] T 3612cd35d8fa4300a1132fe43a63e6c1 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:18.100241 8764 raft_consensus.cc:348] T b6cb5119a6db403ba3cae9fa9b121211 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:18.100294 8762 raft_consensus.cc:348] T 3612cd35d8fa4300a1132fe43a63e6c1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:18.100358 8764 raft_consensus.cc:374] T b6cb5119a6db403ba3cae9fa9b121211 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:18.100387 8762 raft_consensus.cc:374] T 3612cd35d8fa4300a1132fe43a63e6c1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:18.100389 8764 raft_consensus.cc:729] T b6cb5119a6db403ba3cae9fa9b121211 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:18.100418 8762 raft_consensus.cc:729] T 3612cd35d8fa4300a1132fe43a63e6c1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:18.100515 8762 consensus_queue.cc:260] T 3612cd35d8fa4300a1132fe43a63e6c1 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:18.100515 8764 consensus_queue.cc:260] T b6cb5119a6db403ba3cae9fa9b121211 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:18.100795 8764 ts_tablet_manager.cc:1398] T b6cb5119a6db403ba3cae9fa9b121211 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:18.101095 8762 ts_tablet_manager.cc:1398] T 3612cd35d8fa4300a1132fe43a63e6c1 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:18.101291 8763 tablet_bootstrap.cc:492] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:18.101377 8763 ts_tablet_manager.cc:1367] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240503 13:59:18.101797 8763 raft_consensus.cc:348] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:18.101909 8763 raft_consensus.cc:374] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:18.101961 8763 raft_consensus.cc:729] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:18.102077 8763 consensus_queue.cc:260] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:18.102416 8763 ts_tablet_manager.cc:1398] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:18.102614 8764 tablet_bootstrap.cc:492] T f9d875f4872e4f5da12a326381d523c0 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:18.103459 8764 tablet_bootstrap.cc:654] T f9d875f4872e4f5da12a326381d523c0 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:18.103462 8763 tablet_bootstrap.cc:492] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:18.104409 8763 tablet_bootstrap.cc:654] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:18.104563 8764 tablet_bootstrap.cc:492] T f9d875f4872e4f5da12a326381d523c0 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:18.104647 8764 ts_tablet_manager.cc:1367] T f9d875f4872e4f5da12a326381d523c0 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:18.105065 8764 raft_consensus.cc:348] T f9d875f4872e4f5da12a326381d523c0 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:18.105167 8764 raft_consensus.cc:374] T f9d875f4872e4f5da12a326381d523c0 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:18.105198 8764 raft_consensus.cc:729] T f9d875f4872e4f5da12a326381d523c0 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:18.105312 8764 consensus_queue.cc:260] T f9d875f4872e4f5da12a326381d523c0 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:18.105612 8764 ts_tablet_manager.cc:1398] T f9d875f4872e4f5da12a326381d523c0 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:18.105754 8764 tablet_bootstrap.cc:492] T 3612cd35d8fa4300a1132fe43a63e6c1 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:18.106616 8764 tablet_bootstrap.cc:654] T 3612cd35d8fa4300a1132fe43a63e6c1 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:18.106875 8767 tablet_bootstrap.cc:492] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:18.106956 8767 ts_tablet_manager.cc:1367] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 13:59:18.107532 8763 tablet_bootstrap.cc:492] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:18.107612 8763 ts_tablet_manager.cc:1367] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 13:59:18.107703 8764 tablet_bootstrap.cc:492] T 3612cd35d8fa4300a1132fe43a63e6c1 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:18.107770 8764 ts_tablet_manager.cc:1367] T 3612cd35d8fa4300a1132fe43a63e6c1 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:18.108016 8763 raft_consensus.cc:348] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:18.108003 8767 raft_consensus.cc:348] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:18.108110 8763 raft_consensus.cc:374] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:18.108134 8767 raft_consensus.cc:374] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:18.108143 8763 raft_consensus.cc:729] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:18.108130 8764 raft_consensus.cc:348] T 3612cd35d8fa4300a1132fe43a63e6c1 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:18.108173 8767 raft_consensus.cc:729] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:18.108203 8764 raft_consensus.cc:374] T 3612cd35d8fa4300a1132fe43a63e6c1 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:18.108224 8764 raft_consensus.cc:729] T 3612cd35d8fa4300a1132fe43a63e6c1 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:18.108285 8763 consensus_queue.cc:260] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:18.108284 8767 consensus_queue.cc:260] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:18.108335 8764 consensus_queue.cc:260] T 3612cd35d8fa4300a1132fe43a63e6c1 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:18.108525 8763 ts_tablet_manager.cc:1398] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:18.108546 8767 ts_tablet_manager.cc:1398] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:18.108565 8764 ts_tablet_manager.cc:1398] T 3612cd35d8fa4300a1132fe43a63e6c1 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:18.108707 8767 tablet_bootstrap.cc:492] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:18.109661 8767 tablet_bootstrap.cc:654] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:18.115334 8767 tablet_bootstrap.cc:492] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:18.115680 8767 ts_tablet_manager.cc:1367] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:59:18.116266 8767 raft_consensus.cc:348] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:18.116384 8767 raft_consensus.cc:374] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:18.116418 8767 raft_consensus.cc:729] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:18.116570 8767 consensus_queue.cc:260] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:18.116897 8767 ts_tablet_manager.cc:1398] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:18.171006 8656 raft_consensus.cc:482] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:18.171147 8656 raft_consensus.cc:504] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:18.171553 8656 leader_election.cc:288] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:18.171918 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9d875f4872e4f5da12a326381d523c0" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:18.171941 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9d875f4872e4f5da12a326381d523c0" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:18.172050 6921 raft_consensus.cc:2454] T f9d875f4872e4f5da12a326381d523c0 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:18.172070 6791 raft_consensus.cc:2454] T f9d875f4872e4f5da12a326381d523c0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:18.172348 6464 leader_election.cc:302] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:18.172510 8656 raft_consensus.cc:2781] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:18.172571 8656 raft_consensus.cc:482] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:18.172607 8656 raft_consensus.cc:3037] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:18.173462 8656 raft_consensus.cc:504] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:18.173750 8656 leader_election.cc:288] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:18.174062 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9d875f4872e4f5da12a326381d523c0" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:18.174072 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9d875f4872e4f5da12a326381d523c0" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:18.174166 6921 raft_consensus.cc:3037] T f9d875f4872e4f5da12a326381d523c0 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:18.174167 6791 raft_consensus.cc:3037] T f9d875f4872e4f5da12a326381d523c0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:18.174916 6921 raft_consensus.cc:2454] T f9d875f4872e4f5da12a326381d523c0 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:18.174929 6791 raft_consensus.cc:2454] T f9d875f4872e4f5da12a326381d523c0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:18.175158 6464 leader_election.cc:302] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:18.175388 8656 raft_consensus.cc:2781] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:18.175478 8656 raft_consensus.cc:686] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Becoming Leader. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Running, Role: LEADER
I20240503 13:59:18.175629 8656 consensus_queue.cc:237] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:18.177824 6382 catalog_manager.cc:5484] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:18.191175 8754 raft_consensus.cc:482] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:18.191336 8754 raft_consensus.cc:504] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:18.191741 8754 leader_election.cc:288] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:18.192098 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6cb5119a6db403ba3cae9fa9b121211" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:18.192098 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6cb5119a6db403ba3cae9fa9b121211" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:18.192220 6528 raft_consensus.cc:2454] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:18.192220 6921 raft_consensus.cc:2454] T b6cb5119a6db403ba3cae9fa9b121211 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:18.192498 6595 leader_election.cc:302] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:18.192670 8754 raft_consensus.cc:2781] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:18.192731 8754 raft_consensus.cc:482] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:18.192785 8754 raft_consensus.cc:3037] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:18.193578 8754 raft_consensus.cc:504] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:18.193873 8754 leader_election.cc:288] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:18.194160 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6cb5119a6db403ba3cae9fa9b121211" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:18.194209 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6cb5119a6db403ba3cae9fa9b121211" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:18.194275 6921 raft_consensus.cc:3037] T b6cb5119a6db403ba3cae9fa9b121211 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:18.194311 6528 raft_consensus.cc:3037] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:18.195238 6921 raft_consensus.cc:2454] T b6cb5119a6db403ba3cae9fa9b121211 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:18.195235 6528 raft_consensus.cc:2454] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:18.195575 6595 leader_election.cc:302] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:18.195757 8754 raft_consensus.cc:2781] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:18.195848 8754 raft_consensus.cc:686] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 13:59:18.195983 8754 consensus_queue.cc:237] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:18.197862 6382 catalog_manager.cc:5484] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:18.273581 8754 raft_consensus.cc:482] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:18.273705 8754 raft_consensus.cc:504] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:18.274036 8754 leader_election.cc:288] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:18.274370 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3612cd35d8fa4300a1132fe43a63e6c1" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:18.274389 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3612cd35d8fa4300a1132fe43a63e6c1" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:18.274495 6922 raft_consensus.cc:2454] T 3612cd35d8fa4300a1132fe43a63e6c1 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:18.274508 6791 raft_consensus.cc:2454] T 3612cd35d8fa4300a1132fe43a63e6c1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:18.274785 6594 leader_election.cc:302] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:18.274936 8754 raft_consensus.cc:2781] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:18.274992 8754 raft_consensus.cc:482] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:18.275018 8754 raft_consensus.cc:3037] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:18.275852 8754 raft_consensus.cc:504] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:18.276131 8754 leader_election.cc:288] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:18.276408 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3612cd35d8fa4300a1132fe43a63e6c1" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:18.276494 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3612cd35d8fa4300a1132fe43a63e6c1" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:18.276546 6922 raft_consensus.cc:3037] T 3612cd35d8fa4300a1132fe43a63e6c1 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:18.276607 6791 raft_consensus.cc:3037] T 3612cd35d8fa4300a1132fe43a63e6c1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:18.277493 6922 raft_consensus.cc:2454] T 3612cd35d8fa4300a1132fe43a63e6c1 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:18.277493 6791 raft_consensus.cc:2454] T 3612cd35d8fa4300a1132fe43a63e6c1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:18.277758 6595 leader_election.cc:302] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:18.277949 8754 raft_consensus.cc:2781] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:18.278026 8754 raft_consensus.cc:686] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 13:59:18.278173 8754 consensus_queue.cc:237] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:18.280031 6382 catalog_manager.cc:5484] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:18.393936 8828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:18.394193 8828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:18.397408 8828 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_19'
I20240503 13:59:18.482484 6922 raft_consensus.cc:1261] T 3612cd35d8fa4300a1132fe43a63e6c1 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:18.483016 8754 consensus_queue.cc:1035] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:18.483819 6528 raft_consensus.cc:1261] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:18.483933 6791 raft_consensus.cc:1261] T 3612cd35d8fa4300a1132fe43a63e6c1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:18.484478 8423 consensus_queue.cc:1035] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:18.484613 6791 raft_consensus.cc:1261] T f9d875f4872e4f5da12a326381d523c0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:18.484652 6922 raft_consensus.cc:1261] T f9d875f4872e4f5da12a326381d523c0 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:18.485049 8656 consensus_queue.cc:1035] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:18.487951 8754 consensus_queue.cc:1035] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:18.490069 6920 raft_consensus.cc:1261] T b6cb5119a6db403ba3cae9fa9b121211 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:18.490309 8656 consensus_queue.cc:1035] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:18.493980 8423 consensus_queue.cc:1035] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 120.059 ms
time per row: 0.0600295 ms
I20240503 13:59:18.563050 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:34547
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:45485
--webserver_port=37125
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 13:59:18.674181 8866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:18.674393 8866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:18.674443 8866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:18.677390 8866 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:18.677453 8866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:18.677534 8866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 13:59:18.681741 8866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:34547
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=37125
--tserver_master_addrs=127.0.99.190:45485
--never_fsync=true
--heap_profile_path=/tmp/kudu.8866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:18.682615 8866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:18.683732 8866 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:18.693174 8871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:18.693922 8874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:18.694715 8872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:18.695566 8866 server_base.cc:1034] running on GCE node
I20240503 13:59:18.695945 8866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:18.696498 8866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:18.697654 8866 hybrid_clock.cc:648] HybridClock initialized: now 1714744758697624 us; error 43 us; skew 500 ppm
I20240503 13:59:18.699704 8866 webserver.cc:460] Webserver started at http://127.0.99.133:37125/ using document root <none> and password file <none>
I20240503 13:59:18.700464 8866 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:18.700598 8866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:18.703977 8866 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:59:18.705986 8880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:18.707144 8866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:18.707258 8866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "06ca8dbc004c42b99faa527a4e6f9545"
format_stamp: "Formatted at 2024-05-03 13:59:03 on dist-test-slave-fr9h"
I20240503 13:59:18.707695 8866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:18.720692 8866 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:18.721210 8866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:18.721366 8866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:18.721901 8866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:18.722831 8866 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:18.722878 8866 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:18.722932 8866 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:18.722967 8866 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:18.731241 8866 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:34547
I20240503 13:59:18.731307 8993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:34547 every 8 connection(s)
I20240503 13:59:18.732391 8866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 13:59:18.738654 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 8866
I20240503 13:59:18.739076 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:39481
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:45485
--webserver_port=45163
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:18.740804 8994 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45485
I20240503 13:59:18.741053 8994 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:18.741617 8994 heartbeater.cc:511] Master 127.0.99.190:45485 requested a full tablet report, sending...
I20240503 13:59:18.742653 6382 ts_manager.cc:194] Re-registered known tserver with Master: 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:18.743182 6382 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:45423
W20240503 13:59:18.853964 8997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:18.854219 8997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:18.854277 8997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:18.857656 8997 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:18.857725 8997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:18.857810 8997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 13:59:18.861341 8997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:39481
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=45163
--tserver_master_addrs=127.0.99.190:45485
--never_fsync=true
--heap_profile_path=/tmp/kudu.8997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:18.862210 8997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:18.863366 8997 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:18.871918 9007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:18.872682 9004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:18.873540 9005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:18.874239 8997 server_base.cc:1034] running on GCE node
I20240503 13:59:18.874670 8997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:18.875203 8997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:18.876375 8997 hybrid_clock.cc:648] HybridClock initialized: now 1714744758876345 us; error 43 us; skew 500 ppm
I20240503 13:59:18.878275 8997 webserver.cc:460] Webserver started at http://127.0.99.134:45163/ using document root <none> and password file <none>
I20240503 13:59:18.878829 8997 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:18.878902 8997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:18.882097 8997 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:18.884013 9013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:18.885036 8997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:18.885133 8997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "8d574fb4a89d4eea947c859470d44dd4"
format_stamp: "Formatted at 2024-05-03 13:59:03 on dist-test-slave-fr9h"
I20240503 13:59:18.885540 8997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:18.905130 8997 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:18.905756 8997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:18.905916 8997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:18.906455 8997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:18.907456 8997 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:18.907509 8997 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:18.907569 8997 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:18.907593 8997 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:18.917073 8997 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:39481
I20240503 13:59:18.917117 9126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:39481 every 8 connection(s)
I20240503 13:59:18.918183 8997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 13:59:18.925328 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 8997
I20240503 13:59:18.925770 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:34091
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:45485
--webserver_port=37925
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:18.928130 9127 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45485
I20240503 13:59:18.928390 9127 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:18.929013 9127 heartbeater.cc:511] Master 127.0.99.190:45485 requested a full tablet report, sending...
I20240503 13:59:18.930181 6382 ts_manager.cc:194] Re-registered known tserver with Master: 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:18.931026 6382 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:50365
W20240503 13:59:19.042075 9130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:19.042284 9130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:19.042340 9130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:19.045644 9130 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:19.045704 9130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:19.045790 9130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 13:59:19.049641 9130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:34091
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=37925
--tserver_master_addrs=127.0.99.190:45485
--never_fsync=true
--heap_profile_path=/tmp/kudu.9130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:19.050621 9130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:19.051791 9130 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:19.060652 9136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:19.061458 9138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:19.062250 9135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:19.063159 9130 server_base.cc:1034] running on GCE node
I20240503 13:59:19.063627 9130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:19.064261 9130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:19.065454 9130 hybrid_clock.cc:648] HybridClock initialized: now 1714744759065429 us; error 42 us; skew 500 ppm
I20240503 13:59:19.067351 9130 webserver.cc:460] Webserver started at http://127.0.99.135:37925/ using document root <none> and password file <none>
I20240503 13:59:19.067881 9130 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:19.067950 9130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:19.071657 9130 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20240503 13:59:19.073616 9144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:19.074642 9130 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:19.074759 9130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "ab1d21935d6049888eb0ceccc3c9d735"
format_stamp: "Formatted at 2024-05-03 13:59:03 on dist-test-slave-fr9h"
I20240503 13:59:19.075206 9130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:19.087728 9130 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:19.088250 9130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:19.088435 9130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:19.088984 9130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:19.089859 9130 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:19.089913 9130 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:19.089982 9130 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:19.090021 9130 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:19.099344 9130 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:34091
I20240503 13:59:19.099383 9257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:34091 every 8 connection(s)
I20240503 13:59:19.100318 9130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 13:59:19.101850 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 9130
I20240503 13:59:19.102267 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
/tmp/dist-test-taskSuJMqo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:41931
--local_ip_for_outbound_sockets=127.0.99.136
--tserver_master_addrs=127.0.99.190:45485
--webserver_port=36929
--webserver_interface=127.0.99.136
--builtin_ntp_servers=127.0.99.148:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:19.110338 9258 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45485
I20240503 13:59:19.110544 9258 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:19.111073 9258 heartbeater.cc:511] Master 127.0.99.190:45485 requested a full tablet report, sending...
I20240503 13:59:19.112150 6382 ts_manager.cc:194] Re-registered known tserver with Master: ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:19.112874 6382 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:53707
W20240503 13:59:19.212447 9261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:19.212697 9261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:19.212764 9261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:19.215862 9261 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:19.215919 9261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:19.215984 9261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 13:59:19.219907 9261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:41931
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=36929
--tserver_master_addrs=127.0.99.190:45485
--never_fsync=true
--heap_profile_path=/tmp/kudu.9261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:19.220917 9261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:19.222169 9261 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:19.232189 9267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:19.233130 9269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:19.233960 9266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:19.234746 9261 server_base.cc:1034] running on GCE node
I20240503 13:59:19.235114 9261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:19.235697 9261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:19.236848 9261 hybrid_clock.cc:648] HybridClock initialized: now 1714744759236833 us; error 29 us; skew 500 ppm
I20240503 13:59:19.239006 9261 webserver.cc:460] Webserver started at http://127.0.99.136:36929/ using document root <none> and password file <none>
I20240503 13:59:19.239621 9261 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:19.239688 9261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:19.242866 9261 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 13:59:19.244659 9275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:19.245625 9261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:19.245712 9261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "0d77752181e944698e809e325e5937e6"
format_stamp: "Formatted at 2024-05-03 13:59:03 on dist-test-slave-fr9h"
I20240503 13:59:19.246104 9261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:19.267858 9261 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:19.268429 9261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:19.268590 9261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:19.269086 9261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:19.269932 9261 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:19.269977 9261 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:19.270020 9261 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:19.270056 9261 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:19.278297 9261 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:41931
I20240503 13:59:19.278326 9388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:41931 every 8 connection(s)
I20240503 13:59:19.279383 9261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 13:59:19.288229 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSuJMqo/build/debug/bin/kudu as pid 9261
I20240503 13:59:19.288820 9389 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45485
I20240503 13:59:19.289069 9389 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:19.289595 9389 heartbeater.cc:511] Master 127.0.99.190:45485 requested a full tablet report, sending...
I20240503 13:59:19.290642 6382 ts_manager.cc:194] Re-registered known tserver with Master: 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:19.291307 6387 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:19.291383 6382 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:46905
I20240503 13:59:19.305661 9323 tablet_service.cc:1467] Processing CreateTablet for tablet bf1d935ef9594bedad09f8cd092e7290 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b1afb70a06f34e98bb586a981218e83f]), partition=
I20240503 13:59:19.306362 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf1d935ef9594bedad09f8cd092e7290. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:19.311852 9405 tablet_bootstrap.cc:492] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:19.313684 9405 tablet_bootstrap.cc:654] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:19.314390 9405 log.cc:824] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:19.316226 9405 tablet_bootstrap.cc:492] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:19.316411 9405 ts_tablet_manager.cc:1367] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 13:59:19.319005 9191 tablet_service.cc:1467] Processing CreateTablet for tablet bf1d935ef9594bedad09f8cd092e7290 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b1afb70a06f34e98bb586a981218e83f]), partition=
I20240503 13:59:19.319831 9191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf1d935ef9594bedad09f8cd092e7290. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:19.320237 9405 raft_consensus.cc:348] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:19.320411 9405 raft_consensus.cc:374] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:19.320454 9405 raft_consensus.cc:729] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:19.320863 9405 consensus_queue.cc:260] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:19.321588 9405 ts_tablet_manager.cc:1398] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 13:59:19.321893 9389 heartbeater.cc:503] Master 127.0.99.190:45485 was elected leader, sending a full tablet report...
I20240503 13:59:19.326334 9409 tablet_bootstrap.cc:492] T bf1d935ef9594bedad09f8cd092e7290 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:19.328083 9409 tablet_bootstrap.cc:654] T bf1d935ef9594bedad09f8cd092e7290 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:19.328938 9409 log.cc:824] T bf1d935ef9594bedad09f8cd092e7290 P ab1d21935d6049888eb0ceccc3c9d735: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:19.332017 9409 tablet_bootstrap.cc:492] T bf1d935ef9594bedad09f8cd092e7290 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:19.332178 9409 ts_tablet_manager.cc:1367] T bf1d935ef9594bedad09f8cd092e7290 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 13:59:19.332792 8928 tablet_service.cc:1467] Processing CreateTablet for tablet bf1d935ef9594bedad09f8cd092e7290 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b1afb70a06f34e98bb586a981218e83f]), partition=
I20240503 13:59:19.333487 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf1d935ef9594bedad09f8cd092e7290. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:19.335700 9409 raft_consensus.cc:348] T bf1d935ef9594bedad09f8cd092e7290 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:19.335901 9409 raft_consensus.cc:374] T bf1d935ef9594bedad09f8cd092e7290 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:19.335965 9409 raft_consensus.cc:729] T bf1d935ef9594bedad09f8cd092e7290 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:19.336422 9409 consensus_queue.cc:260] T bf1d935ef9594bedad09f8cd092e7290 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:19.337249 9409 ts_tablet_manager.cc:1398] T bf1d935ef9594bedad09f8cd092e7290 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 13:59:19.337594 9258 heartbeater.cc:503] Master 127.0.99.190:45485 was elected leader, sending a full tablet report...
I20240503 13:59:19.341907 9412 tablet_bootstrap.cc:492] T bf1d935ef9594bedad09f8cd092e7290 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:19.344069 9412 tablet_bootstrap.cc:654] T bf1d935ef9594bedad09f8cd092e7290 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:19.345114 9412 log.cc:824] T bf1d935ef9594bedad09f8cd092e7290 P 06ca8dbc004c42b99faa527a4e6f9545: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:19.346518 9412 tablet_bootstrap.cc:492] T bf1d935ef9594bedad09f8cd092e7290 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:19.346697 9412 ts_tablet_manager.cc:1367] T bf1d935ef9594bedad09f8cd092e7290 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 13:59:19.349571 9412 raft_consensus.cc:348] T bf1d935ef9594bedad09f8cd092e7290 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:19.349740 9412 raft_consensus.cc:374] T bf1d935ef9594bedad09f8cd092e7290 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:19.349810 9412 raft_consensus.cc:729] T bf1d935ef9594bedad09f8cd092e7290 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:19.350186 9412 consensus_queue.cc:260] T bf1d935ef9594bedad09f8cd092e7290 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:19.350876 8994 heartbeater.cc:503] Master 127.0.99.190:45485 was elected leader, sending a full tablet report...
W20240503 13:59:19.351096 9259 tablet.cc:2369] T bf1d935ef9594bedad09f8cd092e7290 P ab1d21935d6049888eb0ceccc3c9d735: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:59:19.350951 9412 ts_tablet_manager.cc:1398] T bf1d935ef9594bedad09f8cd092e7290 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240503 13:59:19.425712 9408 raft_consensus.cc:482] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:19.425887 9408 raft_consensus.cc:504] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:19.427090 9408 leader_election.cc:288] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:19.432618 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf1d935ef9594bedad09f8cd092e7290" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:19.432904 9211 raft_consensus.cc:2454] T bf1d935ef9594bedad09f8cd092e7290 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:19.433351 9278 leader_election.cc:302] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:19.433427 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf1d935ef9594bedad09f8cd092e7290" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:19.433604 9408 raft_consensus.cc:2781] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:19.433684 9408 raft_consensus.cc:482] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:19.433730 9408 raft_consensus.cc:3037] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:19.433707 8948 raft_consensus.cc:2454] T bf1d935ef9594bedad09f8cd092e7290 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:19.434851 9408 raft_consensus.cc:504] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:19.435217 9408 leader_election.cc:288] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:19.435564 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf1d935ef9594bedad09f8cd092e7290" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:19.435626 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf1d935ef9594bedad09f8cd092e7290" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:19.435694 9211 raft_consensus.cc:3037] T bf1d935ef9594bedad09f8cd092e7290 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:19.435781 8948 raft_consensus.cc:3037] T bf1d935ef9594bedad09f8cd092e7290 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:19.436762 9211 raft_consensus.cc:2454] T bf1d935ef9594bedad09f8cd092e7290 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:19.436762 8948 raft_consensus.cc:2454] T bf1d935ef9594bedad09f8cd092e7290 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:19.437019 9278 leader_election.cc:302] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:19.443408 9408 raft_consensus.cc:2781] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:19.443651 9408 raft_consensus.cc:686] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:19.443984 9408 consensus_queue.cc:237] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:19.447083 6387 catalog_manager.cc:5484] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:19.483675 8995 tablet.cc:2369] T bf1d935ef9594bedad09f8cd092e7290 P 06ca8dbc004c42b99faa527a4e6f9545: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 13:59:19.530251 9390 tablet.cc:2369] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:59:19.540820 8928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:19.542899 6639 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:59:19.543318 9323 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:19.550134 9191 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:19.552819 6508 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:59:19.553490 9058 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:19.568267 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:19.576826 8948 raft_consensus.cc:1261] T bf1d935ef9594bedad09f8cd092e7290 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:19.578199 9408 consensus_queue.cc:1035] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:19.586117 6379 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet bf1d935ef9594bedad09f8cd092e7290 (table rebalancer_dynamic_table [id=b1afb70a06f34e98bb586a981218e83f]). Expected version 1 got 0
I20240503 13:59:19.587785 6769 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:59:19.588516 9211 raft_consensus.cc:1261] T bf1d935ef9594bedad09f8cd092e7290 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:19.589159 9408 consensus_queue.cc:1035] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:19.592679 6902 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:59:19.593456 6378 catalog_manager.cc:5576] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091) does not have the latest schema for tablet bf1d935ef9594bedad09f8cd092e7290 (table rebalancer_dynamic_table [id=b1afb70a06f34e98bb586a981218e83f]). Expected version 1 got 0
I20240503 13:59:19.597798 6378 catalog_manager.cc:5576] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931) does not have the latest schema for tablet bf1d935ef9594bedad09f8cd092e7290 (table rebalancer_dynamic_table [id=b1afb70a06f34e98bb586a981218e83f]). Expected version 1 got 0
I20240503 13:59:19.599570 9476 mvcc.cc:205] Tried to move back new op lower bound from 7023594535218860032 to 7023594534685388800. Current Snapshot: MvccSnapshot[applied={T|T < 7023594535218860032}]
I20240503 13:59:19.599714 9480 tablet.cc:1722] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:19.599901 6379 catalog_manager.cc:5576] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931) does not have the latest schema for tablet bf1d935ef9594bedad09f8cd092e7290 (table rebalancer_dynamic_table [id=b1afb70a06f34e98bb586a981218e83f]). Expected version 1 got 0
I20240503 13:59:19.600214 9482 tablet.cc:1722] T bf1d935ef9594bedad09f8cd092e7290 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:19.600525 9480 tablet.cc:1620] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:19.600942 9482 tablet.cc:1620] T bf1d935ef9594bedad09f8cd092e7290 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:19.600965 9478 mvcc.cc:205] Tried to move back new op lower bound from 7023594535218860032 to 7023594534685388800. Current Snapshot: MvccSnapshot[applied={T|T < 7023594535218860032}]
I20240503 13:59:19.601363 6379 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet bf1d935ef9594bedad09f8cd092e7290 (table rebalancer_dynamic_table [id=b1afb70a06f34e98bb586a981218e83f]). Expected version 1 got 0
I20240503 13:59:19.606385 6383 catalog_manager.cc:5576] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091) does not have the latest schema for tablet bf1d935ef9594bedad09f8cd092e7290 (table rebalancer_dynamic_table [id=b1afb70a06f34e98bb586a981218e83f]). Expected version 1 got 0
I20240503 13:59:19.606581 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=b1afb70a06f34e98bb586a981218e83f] alter complete (version 1)
I20240503 13:59:19.614787 9481 tablet.cc:1722] T bf1d935ef9594bedad09f8cd092e7290 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:19.615599 9481 tablet.cc:1620] T bf1d935ef9594bedad09f8cd092e7290 P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:19.710318 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:19.716156 9480 tablet.cc:1722] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:19.716413 9482 tablet.cc:1722] T bf1d935ef9594bedad09f8cd092e7290 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:19.716506 9480 tablet.cc:1620] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:19.716743 9482 tablet.cc:1620] T bf1d935ef9594bedad09f8cd092e7290 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:19.720909 9481 tablet.cc:1722] T bf1d935ef9594bedad09f8cd092e7290 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:19.721200 9481 tablet.cc:1620] T bf1d935ef9594bedad09f8cd092e7290 P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:19.721793 6385 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=b1afb70a06f34e98bb586a981218e83f] alter complete (version 2)
I20240503 13:59:19.739214 8928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:19.746727 9323 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:19.749766 6639 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:59:19.755172 6769 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:59:19.760962 6508 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:59:19.786710 9191 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:19.786707 6902 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:59:19.787097 9058 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:19.789649 6380 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240503 13:59:19.789794 6380 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240503 13:59:19.792742 6386 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:19.792836 6380 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 7662baa6a6e5478f956632b6e5dd1290
I20240503 13:59:19.793208 6380 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet b50f9df84ee44a5a8b24d70a7c0e76d1
I20240503 13:59:19.793398 6380 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet b0a4be5ebd3246b39b011743d9ae5daa
I20240503 13:59:19.794159 6639 tablet_service.cc:1514] Processing DeleteTablet for tablet 7662baa6a6e5478f956632b6e5dd1290 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:19 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:19.794387 6508 tablet_service.cc:1514] Processing DeleteTablet for tablet 7662baa6a6e5478f956632b6e5dd1290 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:19 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:19.794571 6507 tablet_service.cc:1514] Processing DeleteTablet for tablet b50f9df84ee44a5a8b24d70a7c0e76d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:19 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:19.794574 6769 tablet_service.cc:1514] Processing DeleteTablet for tablet b50f9df84ee44a5a8b24d70a7c0e76d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:19 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:19.794159 6640 tablet_service.cc:1514] Processing DeleteTablet for tablet b50f9df84ee44a5a8b24d70a7c0e76d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:19 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:19.794739 6509 tablet_service.cc:1514] Processing DeleteTablet for tablet b0a4be5ebd3246b39b011743d9ae5daa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:19 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:19.794912 6771 tablet_service.cc:1514] Processing DeleteTablet for tablet b0a4be5ebd3246b39b011743d9ae5daa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:19 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:19.795061 6902 tablet_service.cc:1514] Processing DeleteTablet for tablet 7662baa6a6e5478f956632b6e5dd1290 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:19 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:19.795099 6900 tablet_service.cc:1514] Processing DeleteTablet for tablet b0a4be5ebd3246b39b011743d9ae5daa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:19 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:19.796304 9541 tablet_replica.cc:330] T 7662baa6a6e5478f956632b6e5dd1290 P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:19.796676 9541 raft_consensus.cc:2229] T 7662baa6a6e5478f956632b6e5dd1290 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:19.796918 9541 raft_consensus.cc:2258] T 7662baa6a6e5478f956632b6e5dd1290 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:19.797971 9541 ts_tablet_manager.cc:1875] T 7662baa6a6e5478f956632b6e5dd1290 P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:19.800725 9541 ts_tablet_manager.cc:1888] T 7662baa6a6e5478f956632b6e5dd1290 P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:19.800892 9541 log.cc:1196] T 7662baa6a6e5478f956632b6e5dd1290 P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/7662baa6a6e5478f956632b6e5dd1290
I20240503 13:59:19.801275 9541 ts_tablet_manager.cc:1909] T 7662baa6a6e5478f956632b6e5dd1290 P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:19.802569 9541 tablet_replica.cc:330] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:19.803023 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 7662baa6a6e5478f956632b6e5dd1290 (table rebalance_test_table_0 [id=41d8e6d525344c1dabeb32421a3ae38f]) successfully deleted
I20240503 13:59:19.803005 6509 tablet_service.cc:1467] Processing CreateTablet for tablet a9e87083385d4e6bac9bda95670b68a8 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=0be76966adde4247b7969f0f470308fd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:19.803333 6509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e87083385d4e6bac9bda95670b68a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:19.804325 9191 tablet_service.cc:1467] Processing CreateTablet for tablet a9e87083385d4e6bac9bda95670b68a8 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=0be76966adde4247b7969f0f470308fd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:19.804719 9191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e87083385d4e6bac9bda95670b68a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:19.805940 8927 tablet_service.cc:1467] Processing CreateTablet for tablet 873f3615984646159e1b05178c933e18 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=0be76966adde4247b7969f0f470308fd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:19.806270 8927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 873f3615984646159e1b05178c933e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:19.806759 8928 tablet_service.cc:1467] Processing CreateTablet for tablet a9e87083385d4e6bac9bda95670b68a8 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=0be76966adde4247b7969f0f470308fd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:19.806953 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e87083385d4e6bac9bda95670b68a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:19.807212 6900 tablet_service.cc:1467] Processing CreateTablet for tablet 873f3615984646159e1b05178c933e18 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=0be76966adde4247b7969f0f470308fd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:19.807514 6900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 873f3615984646159e1b05178c933e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:19.808673 9192 tablet_service.cc:1467] Processing CreateTablet for tablet 8f3c05440fea43689757d5c5bce8424d (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=0be76966adde4247b7969f0f470308fd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:19.808883 9192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3c05440fea43689757d5c5bce8424d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:19.810860 9409 tablet_bootstrap.cc:492] T a9e87083385d4e6bac9bda95670b68a8 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:19.811069 9542 tablet_replica.cc:330] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:19.811391 9542 raft_consensus.cc:2229] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:19.811715 9542 raft_consensus.cc:2258] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:19.811864 9409 tablet_bootstrap.cc:654] T a9e87083385d4e6bac9bda95670b68a8 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:19.812001 9545 tablet_bootstrap.cc:492] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:19.813356 6771 tablet_service.cc:1467] Processing CreateTablet for tablet 8f3c05440fea43689757d5c5bce8424d (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=0be76966adde4247b7969f0f470308fd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:19.813606 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3c05440fea43689757d5c5bce8424d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:19.814527 9543 tablet_replica.cc:330] T b0a4be5ebd3246b39b011743d9ae5daa P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:19.814790 9543 raft_consensus.cc:2229] T b0a4be5ebd3246b39b011743d9ae5daa P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:19.814930 9543 raft_consensus.cc:2258] T b0a4be5ebd3246b39b011743d9ae5daa P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:19.815829 9545 tablet_bootstrap.cc:654] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:19.815829 9412 tablet_bootstrap.cc:492] T 873f3615984646159e1b05178c933e18 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:19.814332 9542 ts_tablet_manager.cc:1875] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:19.816552 9412 tablet_bootstrap.cc:654] T 873f3615984646159e1b05178c933e18 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:19.816751 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 8f3c05440fea43689757d5c5bce8424d (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=0be76966adde4247b7969f0f470308fd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:19.817732 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3c05440fea43689757d5c5bce8424d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:19.818150 9541 raft_consensus.cc:2229] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:19.818343 9541 raft_consensus.cc:2258] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:19.818553 9061 tablet_service.cc:1467] Processing CreateTablet for tablet 873f3615984646159e1b05178c933e18 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=0be76966adde4247b7969f0f470308fd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:19.818818 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 873f3615984646159e1b05178c933e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:19.819015 9541 ts_tablet_manager.cc:1875] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:19.820273 9543 ts_tablet_manager.cc:1875] T b0a4be5ebd3246b39b011743d9ae5daa P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:19.821054 9541 ts_tablet_manager.cc:1888] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:19.821115 9541 log.cc:1196] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/b50f9df84ee44a5a8b24d70a7c0e76d1
I20240503 13:59:19.822003 9409 tablet_bootstrap.cc:492] T a9e87083385d4e6bac9bda95670b68a8 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:19.822100 9409 ts_tablet_manager.cc:1367] T a9e87083385d4e6bac9bda95670b68a8 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 13:59:19.822196 9542 ts_tablet_manager.cc:1888] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:19.822273 9542 log.cc:1196] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/b50f9df84ee44a5a8b24d70a7c0e76d1
I20240503 13:59:19.822590 9409 raft_consensus.cc:348] T a9e87083385d4e6bac9bda95670b68a8 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:19.822692 9409 raft_consensus.cc:374] T a9e87083385d4e6bac9bda95670b68a8 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:19.822721 9409 raft_consensus.cc:729] T a9e87083385d4e6bac9bda95670b68a8 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:19.822835 9409 consensus_queue.cc:260] T a9e87083385d4e6bac9bda95670b68a8 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:19.822965 6386 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:19.823094 9409 ts_tablet_manager.cc:1398] T a9e87083385d4e6bac9bda95670b68a8 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:19.823264 9409 tablet_bootstrap.cc:492] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:19.824060 9409 tablet_bootstrap.cc:654] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:19.827253 9541 ts_tablet_manager.cc:1909] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:19.827898 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet b50f9df84ee44a5a8b24d70a7c0e76d1 (table rebalance_test_table_0 [id=41d8e6d525344c1dabeb32421a3ae38f]) successfully deleted
I20240503 13:59:19.827939 9541 tablet_replica.cc:330] T b0a4be5ebd3246b39b011743d9ae5daa P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:19.828066 9541 raft_consensus.cc:2229] T b0a4be5ebd3246b39b011743d9ae5daa P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:19.828186 9541 raft_consensus.cc:2258] T b0a4be5ebd3246b39b011743d9ae5daa P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:19.828655 9541 ts_tablet_manager.cc:1875] T b0a4be5ebd3246b39b011743d9ae5daa P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:19.829710 9542 ts_tablet_manager.cc:1909] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:19.830475 9541 ts_tablet_manager.cc:1888] T b0a4be5ebd3246b39b011743d9ae5daa P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:19.830536 9541 log.cc:1196] T b0a4be5ebd3246b39b011743d9ae5daa P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/b0a4be5ebd3246b39b011743d9ae5daa
I20240503 13:59:19.830684 9545 tablet_bootstrap.cc:492] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:19.830778 9541 ts_tablet_manager.cc:1909] T b0a4be5ebd3246b39b011743d9ae5daa P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:19.830873 9545 ts_tablet_manager.cc:1367] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 13:59:19.831385 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet b0a4be5ebd3246b39b011743d9ae5daa (table rebalance_test_table_0 [id=41d8e6d525344c1dabeb32421a3ae38f]) successfully deleted
I20240503 13:59:19.831447 9480 tablet.cc:1722] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:19.831897 9409 tablet_bootstrap.cc:492] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:19.831979 9409 ts_tablet_manager.cc:1367] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20240503 13:59:19.832332 9409 raft_consensus.cc:348] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:19.832324 9545 raft_consensus.cc:348] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:19.832445 9409 raft_consensus.cc:374] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:19.832479 9409 raft_consensus.cc:729] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:19.832552 9545 raft_consensus.cc:374] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:19.832576 9409 consensus_queue.cc:260] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:19.832628 9545 raft_consensus.cc:729] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:19.832804 9545 consensus_queue.cc:260] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:19.832880 9409 ts_tablet_manager.cc:1398] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:19.836066 9481 tablet.cc:1722] T bf1d935ef9594bedad09f8cd092e7290 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:19.836560 9482 tablet.cc:1722] T bf1d935ef9594bedad09f8cd092e7290 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:19.836625 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=b1afb70a06f34e98bb586a981218e83f] alter complete (version 3)
I20240503 13:59:19.837913 9545 ts_tablet_manager.cc:1398] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 13:59:19.840245 9543 ts_tablet_manager.cc:1888] T b0a4be5ebd3246b39b011743d9ae5daa P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:19.841418 9542 tablet_replica.cc:330] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:19.841496 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet b50f9df84ee44a5a8b24d70a7c0e76d1 (table rebalance_test_table_0 [id=41d8e6d525344c1dabeb32421a3ae38f]) successfully deleted
I20240503 13:59:19.841564 9543 log.cc:1196] T b0a4be5ebd3246b39b011743d9ae5daa P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/b0a4be5ebd3246b39b011743d9ae5daa
I20240503 13:59:19.841907 9543 ts_tablet_manager.cc:1909] T b0a4be5ebd3246b39b011743d9ae5daa P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:19.842711 9543 tablet_replica.cc:330] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:19.842732 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet b0a4be5ebd3246b39b011743d9ae5daa (table rebalance_test_table_0 [id=41d8e6d525344c1dabeb32421a3ae38f]) successfully deleted
I20240503 13:59:19.842852 9543 raft_consensus.cc:2229] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:19.842893 9552 tablet_bootstrap.cc:492] T 873f3615984646159e1b05178c933e18 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:19.843086 9543 raft_consensus.cc:2258] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:19.843555 9543 ts_tablet_manager.cc:1875] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:19.845256 9543 ts_tablet_manager.cc:1888] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:19.845320 9543 log.cc:1196] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/7662baa6a6e5478f956632b6e5dd1290
I20240503 13:59:19.845552 9543 ts_tablet_manager.cc:1909] T 7662baa6a6e5478f956632b6e5dd1290 P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:19.845790 9542 raft_consensus.cc:2229] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:19.845911 9552 tablet_bootstrap.cc:654] T 873f3615984646159e1b05178c933e18 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:19.846060 9542 raft_consensus.cc:2258] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:19.846235 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet 7662baa6a6e5478f956632b6e5dd1290 (table rebalance_test_table_0 [id=41d8e6d525344c1dabeb32421a3ae38f]) successfully deleted
I20240503 13:59:19.846730 9552 log.cc:824] T 873f3615984646159e1b05178c933e18 P 8d574fb4a89d4eea947c859470d44dd4: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:19.846760 9542 ts_tablet_manager.cc:1875] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:19.848814 9542 ts_tablet_manager.cc:1888] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:19.848879 9542 log.cc:1196] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/b0a4be5ebd3246b39b011743d9ae5daa
I20240503 13:59:19.849097 9542 ts_tablet_manager.cc:1909] T b0a4be5ebd3246b39b011743d9ae5daa P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:19.849747 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet b0a4be5ebd3246b39b011743d9ae5daa (table rebalance_test_table_0 [id=41d8e6d525344c1dabeb32421a3ae38f]) successfully deleted
I20240503 13:59:19.853000 9540 tablet_replica.cc:330] T 7662baa6a6e5478f956632b6e5dd1290 P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:19.853319 9540 raft_consensus.cc:2229] T 7662baa6a6e5478f956632b6e5dd1290 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:19.853502 9540 raft_consensus.cc:2258] T 7662baa6a6e5478f956632b6e5dd1290 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:19.854494 9540 ts_tablet_manager.cc:1875] T 7662baa6a6e5478f956632b6e5dd1290 P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:19.855490 9547 tablet_bootstrap.cc:492] T 8f3c05440fea43689757d5c5bce8424d P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:19.856316 9412 tablet_bootstrap.cc:492] T 873f3615984646159e1b05178c933e18 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:19.856407 9547 tablet_bootstrap.cc:654] T 8f3c05440fea43689757d5c5bce8424d P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:19.856431 9412 ts_tablet_manager.cc:1367] T 873f3615984646159e1b05178c933e18 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.041s user 0.001s sys 0.000s
I20240503 13:59:19.857023 9412 raft_consensus.cc:348] T 873f3615984646159e1b05178c933e18 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:19.857152 9412 raft_consensus.cc:374] T 873f3615984646159e1b05178c933e18 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:19.857190 9412 raft_consensus.cc:729] T 873f3615984646159e1b05178c933e18 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:19.857334 9412 consensus_queue.cc:260] T 873f3615984646159e1b05178c933e18 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:19.857627 9412 ts_tablet_manager.cc:1398] T 873f3615984646159e1b05178c933e18 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:19.857790 9412 tablet_bootstrap.cc:492] T a9e87083385d4e6bac9bda95670b68a8 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:19.858438 9540 ts_tablet_manager.cc:1888] T 7662baa6a6e5478f956632b6e5dd1290 P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:19.858527 9540 log.cc:1196] T 7662baa6a6e5478f956632b6e5dd1290 P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/7662baa6a6e5478f956632b6e5dd1290
I20240503 13:59:19.858736 9412 tablet_bootstrap.cc:654] T a9e87083385d4e6bac9bda95670b68a8 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:19.858819 9540 ts_tablet_manager.cc:1909] T 7662baa6a6e5478f956632b6e5dd1290 P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:19.859968 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet 7662baa6a6e5478f956632b6e5dd1290 (table rebalance_test_table_0 [id=41d8e6d525344c1dabeb32421a3ae38f]) successfully deleted
I20240503 13:59:19.860683 9544 tablet_bootstrap.cc:492] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:19.859925 9540 tablet_replica.cc:330] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:19.860831 9540 raft_consensus.cc:2229] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:19.861069 9540 raft_consensus.cc:2258] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:19.861629 9540 ts_tablet_manager.cc:1875] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:19.861850 9544 tablet_bootstrap.cc:654] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:19.863142 9412 tablet_bootstrap.cc:492] T a9e87083385d4e6bac9bda95670b68a8 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:19.863217 9412 ts_tablet_manager.cc:1367] T a9e87083385d4e6bac9bda95670b68a8 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:19.863521 9544 tablet_bootstrap.cc:492] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:19.863597 9544 ts_tablet_manager.cc:1367] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:19.863691 9412 raft_consensus.cc:348] T a9e87083385d4e6bac9bda95670b68a8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:19.863776 9412 raft_consensus.cc:374] T a9e87083385d4e6bac9bda95670b68a8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:19.863796 9412 raft_consensus.cc:729] T a9e87083385d4e6bac9bda95670b68a8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:19.863898 9552 tablet_bootstrap.cc:492] T 873f3615984646159e1b05178c933e18 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:19.863898 9412 consensus_queue.cc:260] T a9e87083385d4e6bac9bda95670b68a8 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:19.863965 9544 raft_consensus.cc:348] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:19.864070 9544 raft_consensus.cc:374] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:19.864084 9552 ts_tablet_manager.cc:1367] T 873f3615984646159e1b05178c933e18 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240503 13:59:19.864105 9544 raft_consensus.cc:729] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:19.864140 9412 ts_tablet_manager.cc:1398] T a9e87083385d4e6bac9bda95670b68a8 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:19.865538 9544 consensus_queue.cc:260] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:19.866453 9544 ts_tablet_manager.cc:1398] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:19.869951 9552 raft_consensus.cc:348] T 873f3615984646159e1b05178c933e18 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:19.870177 9552 raft_consensus.cc:374] T 873f3615984646159e1b05178c933e18 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:19.870635 9547 tablet_bootstrap.cc:492] T 8f3c05440fea43689757d5c5bce8424d P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:19.870735 9547 ts_tablet_manager.cc:1367] T 8f3c05440fea43689757d5c5bce8424d P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.000s
I20240503 13:59:19.871121 9547 raft_consensus.cc:348] T 8f3c05440fea43689757d5c5bce8424d P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:19.871271 9547 raft_consensus.cc:374] T 8f3c05440fea43689757d5c5bce8424d P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:19.871337 9547 raft_consensus.cc:729] T 8f3c05440fea43689757d5c5bce8424d P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:19.871457 9547 consensus_queue.cc:260] T 8f3c05440fea43689757d5c5bce8424d P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:19.871699 9547 ts_tablet_manager.cc:1398] T 8f3c05440fea43689757d5c5bce8424d P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:19.871815 9552 raft_consensus.cc:729] T 873f3615984646159e1b05178c933e18 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:19.872310 9552 consensus_queue.cc:260] T 873f3615984646159e1b05178c933e18 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:19.873282 9552 ts_tablet_manager.cc:1398] T 873f3615984646159e1b05178c933e18 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240503 13:59:19.873631 9552 tablet_bootstrap.cc:492] T 8f3c05440fea43689757d5c5bce8424d P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:19.873670 9127 heartbeater.cc:503] Master 127.0.99.190:45485 was elected leader, sending a full tablet report...
I20240503 13:59:19.874627 9552 tablet_bootstrap.cc:654] T 8f3c05440fea43689757d5c5bce8424d P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:19.876353 9540 ts_tablet_manager.cc:1888] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:19.876478 9540 log.cc:1196] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/b50f9df84ee44a5a8b24d70a7c0e76d1
I20240503 13:59:19.876742 9540 ts_tablet_manager.cc:1909] T b50f9df84ee44a5a8b24d70a7c0e76d1 P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:19.877571 9552 tablet_bootstrap.cc:492] T 8f3c05440fea43689757d5c5bce8424d P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:19.877689 9552 ts_tablet_manager.cc:1367] T 8f3c05440fea43689757d5c5bce8424d P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:19.878192 9552 raft_consensus.cc:348] T 8f3c05440fea43689757d5c5bce8424d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:19.878315 9552 raft_consensus.cc:374] T 8f3c05440fea43689757d5c5bce8424d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:19.878343 9552 raft_consensus.cc:729] T 8f3c05440fea43689757d5c5bce8424d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:19.878929 9552 consensus_queue.cc:260] T 8f3c05440fea43689757d5c5bce8424d P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:19.879009 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet b50f9df84ee44a5a8b24d70a7c0e76d1 (table rebalance_test_table_0 [id=41d8e6d525344c1dabeb32421a3ae38f]) successfully deleted
I20240503 13:59:19.879369 9552 ts_tablet_manager.cc:1398] T 8f3c05440fea43689757d5c5bce8424d P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:19.901449 8998 raft_consensus.cc:482] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:19.901562 8998 raft_consensus.cc:504] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:19.902014 8998 leader_election.cc:288] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:19.903334 9411 raft_consensus.cc:482] T a9e87083385d4e6bac9bda95670b68a8 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:19.903445 9411 raft_consensus.cc:504] T a9e87083385d4e6bac9bda95670b68a8 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:19.904632 9411 leader_election.cc:288] T a9e87083385d4e6bac9bda95670b68a8 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:19.907289 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e87083385d4e6bac9bda95670b68a8" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:19.907454 9211 raft_consensus.cc:2454] T a9e87083385d4e6bac9bda95670b68a8 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:19.907667 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e87083385d4e6bac9bda95670b68a8" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:19.907800 8948 raft_consensus.cc:2454] T a9e87083385d4e6bac9bda95670b68a8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:19.908071 6464 leader_election.cc:302] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:19.908218 8998 raft_consensus.cc:2781] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:19.908282 8998 raft_consensus.cc:482] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:19.908340 8998 raft_consensus.cc:3037] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:19.909338 8998 raft_consensus.cc:504] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:19.909669 8998 leader_election.cc:288] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Requested vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:19.910243 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e87083385d4e6bac9bda95670b68a8" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:19.910354 8948 raft_consensus.cc:3037] T a9e87083385d4e6bac9bda95670b68a8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:19.911404 8948 raft_consensus.cc:2454] T a9e87083385d4e6bac9bda95670b68a8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:19.911772 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e87083385d4e6bac9bda95670b68a8" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:19.911793 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e87083385d4e6bac9bda95670b68a8" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:19.911907 9211 raft_consensus.cc:3037] T a9e87083385d4e6bac9bda95670b68a8 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:19.911918 6527 raft_consensus.cc:2379] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab1d21935d6049888eb0ceccc3c9d735 in current term 1: Already voted for candidate 5f46e19644fd4f4b825c86ba6af768da in this term.
I20240503 13:59:19.912840 9211 raft_consensus.cc:2454] T a9e87083385d4e6bac9bda95670b68a8 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:19.913337 6464 leader_election.cc:302] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:19.913420 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e87083385d4e6bac9bda95670b68a8" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:19.913574 8948 raft_consensus.cc:2379] T a9e87083385d4e6bac9bda95670b68a8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab1d21935d6049888eb0ceccc3c9d735 in current term 1: Already voted for candidate 5f46e19644fd4f4b825c86ba6af768da in this term.
I20240503 13:59:19.913892 8998 raft_consensus.cc:2781] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:19.913997 8998 raft_consensus.cc:686] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Becoming Leader. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Running, Role: LEADER
I20240503 13:59:19.914139 8998 consensus_queue.cc:237] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:19.915355 9147 leader_election.cc:302] T a9e87083385d4e6bac9bda95670b68a8 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab1d21935d6049888eb0ceccc3c9d735; no voters: 06ca8dbc004c42b99faa527a4e6f9545, 5f46e19644fd4f4b825c86ba6af768da
I20240503 13:59:19.915727 9411 raft_consensus.cc:2726] T a9e87083385d4e6bac9bda95670b68a8 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 13:59:19.916275 6386 catalog_manager.cc:5484] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } }
W20240503 13:59:19.919764 9128 tablet.cc:2369] T 873f3615984646159e1b05178c933e18 P 8d574fb4a89d4eea947c859470d44dd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:59:19.941457 6386 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:19.941591 6386 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:19.945705 6386 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet bf1d935ef9594bedad09f8cd092e7290
I20240503 13:59:19.946410 9323 tablet_service.cc:1514] Processing DeleteTablet for tablet bf1d935ef9594bedad09f8cd092e7290 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:19 UTC) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:19.946553 8928 tablet_service.cc:1514] Processing DeleteTablet for tablet bf1d935ef9594bedad09f8cd092e7290 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:19 UTC) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:19.946785 9192 tablet_service.cc:1514] Processing DeleteTablet for tablet bf1d935ef9594bedad09f8cd092e7290 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:19 UTC) from {username='slave'} at 127.0.0.1:57960
I20240503 13:59:19.947003 9607 tablet_replica.cc:330] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:19.948381 9607 raft_consensus.cc:2229] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:19.948755 9607 raft_consensus.cc:2258] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:19.949391 9608 tablet_replica.cc:330] T bf1d935ef9594bedad09f8cd092e7290 P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:19.949736 9608 raft_consensus.cc:2229] T bf1d935ef9594bedad09f8cd092e7290 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:19.950237 9608 raft_consensus.cc:2258] T bf1d935ef9594bedad09f8cd092e7290 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:19.951319 9608 ts_tablet_manager.cc:1875] T bf1d935ef9594bedad09f8cd092e7290 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:19.952219 9607 ts_tablet_manager.cc:1875] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:19.954202 9608 ts_tablet_manager.cc:1888] T bf1d935ef9594bedad09f8cd092e7290 P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:19.954288 9608 log.cc:1196] T bf1d935ef9594bedad09f8cd092e7290 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/bf1d935ef9594bedad09f8cd092e7290
I20240503 13:59:19.954320 9607 ts_tablet_manager.cc:1888] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:19.954401 9607 log.cc:1196] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/bf1d935ef9594bedad09f8cd092e7290
I20240503 13:59:19.954614 9608 ts_tablet_manager.cc:1909] T bf1d935ef9594bedad09f8cd092e7290 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting consensus metadata
I20240503 13:59:19.954643 9607 ts_tablet_manager.cc:1909] T bf1d935ef9594bedad09f8cd092e7290 P 0d77752181e944698e809e325e5937e6: Deleting consensus metadata
I20240503 13:59:19.955526 9609 tablet_replica.cc:330] T bf1d935ef9594bedad09f8cd092e7290 P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 13:59:19.955843 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet bf1d935ef9594bedad09f8cd092e7290 (table rebalancer_dynamic_table_new_name [id=b1afb70a06f34e98bb586a981218e83f]) successfully deleted
I20240503 13:59:19.955839 9609 raft_consensus.cc:2229] T bf1d935ef9594bedad09f8cd092e7290 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:19.956126 9609 raft_consensus.cc:2258] T bf1d935ef9594bedad09f8cd092e7290 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:19.956430 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet bf1d935ef9594bedad09f8cd092e7290 (table rebalancer_dynamic_table_new_name [id=b1afb70a06f34e98bb586a981218e83f]) successfully deleted
I20240503 13:59:19.957202 9609 ts_tablet_manager.cc:1875] T bf1d935ef9594bedad09f8cd092e7290 P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:19.959228 9609 ts_tablet_manager.cc:1888] T bf1d935ef9594bedad09f8cd092e7290 P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:19.959332 9609 log.cc:1196] T bf1d935ef9594bedad09f8cd092e7290 P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/bf1d935ef9594bedad09f8cd092e7290
I20240503 13:59:19.959650 9609 ts_tablet_manager.cc:1909] T bf1d935ef9594bedad09f8cd092e7290 P ab1d21935d6049888eb0ceccc3c9d735: Deleting consensus metadata
I20240503 13:59:19.960601 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet bf1d935ef9594bedad09f8cd092e7290 (table rebalancer_dynamic_table_new_name [id=b1afb70a06f34e98bb586a981218e83f]) successfully deleted
I20240503 13:59:20.011837 9411 raft_consensus.cc:482] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:20.011950 9411 raft_consensus.cc:504] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:20.012454 9411 leader_election.cc:288] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:20.016407 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3c05440fea43689757d5c5bce8424d" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:20.016566 6791 raft_consensus.cc:2454] T 8f3c05440fea43689757d5c5bce8424d P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:20.016577 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3c05440fea43689757d5c5bce8424d" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:20.016872 9146 leader_election.cc:302] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:20.016891 9081 raft_consensus.cc:2454] T 8f3c05440fea43689757d5c5bce8424d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:20.017052 9411 raft_consensus.cc:2781] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:20.017134 9411 raft_consensus.cc:482] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:20.017186 9411 raft_consensus.cc:3037] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:20.017995 9411 raft_consensus.cc:504] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:20.018308 9411 leader_election.cc:288] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:20.018646 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3c05440fea43689757d5c5bce8424d" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:20.018646 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3c05440fea43689757d5c5bce8424d" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:20.018760 6791 raft_consensus.cc:3037] T 8f3c05440fea43689757d5c5bce8424d P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:20.018766 9081 raft_consensus.cc:3037] T 8f3c05440fea43689757d5c5bce8424d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:20.019948 6791 raft_consensus.cc:2454] T 8f3c05440fea43689757d5c5bce8424d P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:20.019950 9081 raft_consensus.cc:2454] T 8f3c05440fea43689757d5c5bce8424d P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:20.020237 9147 leader_election.cc:302] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:20.020490 9411 raft_consensus.cc:2781] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:20.020641 9411 raft_consensus.cc:686] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 13:59:20.020866 9411 consensus_queue.cc:237] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:20.023130 6379 catalog_manager.cc:5484] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:20.033406 8600 raft_consensus.cc:482] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:20.033524 8600 raft_consensus.cc:504] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:20.033941 8600 leader_election.cc:288] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:20.037668 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873f3615984646159e1b05178c933e18" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:20.037792 9081 raft_consensus.cc:2454] T 873f3615984646159e1b05178c933e18 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:20.038069 6857 leader_election.cc:302] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:20.038220 8600 raft_consensus.cc:2781] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:20.038218 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873f3615984646159e1b05178c933e18" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:20.038281 8600 raft_consensus.cc:482] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:20.038326 8600 raft_consensus.cc:3037] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:20.038367 8948 raft_consensus.cc:2454] T 873f3615984646159e1b05178c933e18 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:20.039252 8600 raft_consensus.cc:504] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:20.039608 8600 leader_election.cc:288] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:20.039918 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873f3615984646159e1b05178c933e18" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:20.040060 9081 raft_consensus.cc:3037] T 873f3615984646159e1b05178c933e18 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:20.040096 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873f3615984646159e1b05178c933e18" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:20.040195 8948 raft_consensus.cc:3037] T 873f3615984646159e1b05178c933e18 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:20.040798 9081 raft_consensus.cc:2454] T 873f3615984646159e1b05178c933e18 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:20.040839 8948 raft_consensus.cc:2454] T 873f3615984646159e1b05178c933e18 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:20.041074 6857 leader_election.cc:302] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:20.041515 8600 raft_consensus.cc:2781] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:20.041597 8600 raft_consensus.cc:686] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:20.041736 8600 consensus_queue.cc:237] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:20.043605 6379 catalog_manager.cc:5484] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:20.047638 6379 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:20.052102 9323 tablet_service.cc:1467] Processing CreateTablet for tablet c868ae9dcad643e98da94752f4de43b8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1ced52dfa09a490fbf1d05b376fc12af]), partition=
I20240503 13:59:20.052165 8928 tablet_service.cc:1467] Processing CreateTablet for tablet c868ae9dcad643e98da94752f4de43b8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1ced52dfa09a490fbf1d05b376fc12af]), partition=
I20240503 13:59:20.052220 9058 tablet_service.cc:1467] Processing CreateTablet for tablet c868ae9dcad643e98da94752f4de43b8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1ced52dfa09a490fbf1d05b376fc12af]), partition=
I20240503 13:59:20.052352 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c868ae9dcad643e98da94752f4de43b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:20.052352 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c868ae9dcad643e98da94752f4de43b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:20.052414 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c868ae9dcad643e98da94752f4de43b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:20.053962 9552 tablet_bootstrap.cc:492] T c868ae9dcad643e98da94752f4de43b8 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:20.054255 9412 tablet_bootstrap.cc:492] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:20.054348 9615 tablet_bootstrap.cc:492] T c868ae9dcad643e98da94752f4de43b8 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:20.054872 9552 tablet_bootstrap.cc:654] T c868ae9dcad643e98da94752f4de43b8 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:20.055212 9615 tablet_bootstrap.cc:654] T c868ae9dcad643e98da94752f4de43b8 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:20.055398 9412 tablet_bootstrap.cc:654] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:20.056203 9552 tablet_bootstrap.cc:492] T c868ae9dcad643e98da94752f4de43b8 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:20.056282 9552 ts_tablet_manager.cc:1367] T c868ae9dcad643e98da94752f4de43b8 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:20.056282 9615 tablet_bootstrap.cc:492] T c868ae9dcad643e98da94752f4de43b8 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:20.056360 9615 ts_tablet_manager.cc:1367] T c868ae9dcad643e98da94752f4de43b8 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:20.056668 9552 raft_consensus.cc:348] T c868ae9dcad643e98da94752f4de43b8 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:20.056754 9552 raft_consensus.cc:374] T c868ae9dcad643e98da94752f4de43b8 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:20.056787 9552 raft_consensus.cc:729] T c868ae9dcad643e98da94752f4de43b8 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:20.056798 9615 raft_consensus.cc:348] T c868ae9dcad643e98da94752f4de43b8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:20.056895 9615 raft_consensus.cc:374] T c868ae9dcad643e98da94752f4de43b8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:20.056928 9615 raft_consensus.cc:729] T c868ae9dcad643e98da94752f4de43b8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:20.056903 9552 consensus_queue.cc:260] T c868ae9dcad643e98da94752f4de43b8 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:20.057168 9552 ts_tablet_manager.cc:1398] T c868ae9dcad643e98da94752f4de43b8 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:20.057209 9615 consensus_queue.cc:260] T c868ae9dcad643e98da94752f4de43b8 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:20.057456 9615 ts_tablet_manager.cc:1398] T c868ae9dcad643e98da94752f4de43b8 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:20.059492 9412 tablet_bootstrap.cc:492] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:20.059595 9412 ts_tablet_manager.cc:1367] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 13:59:20.060042 9412 raft_consensus.cc:348] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:20.060130 9412 raft_consensus.cc:374] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:20.060165 9412 raft_consensus.cc:729] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:20.060289 9412 consensus_queue.cc:260] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:20.060994 9412 ts_tablet_manager.cc:1398] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:20.110215 9414 raft_consensus.cc:482] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:20.110358 9414 raft_consensus.cc:504] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:20.111429 9414 leader_election.cc:288] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:20.115424 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c868ae9dcad643e98da94752f4de43b8" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:20.115592 9081 raft_consensus.cc:2454] T c868ae9dcad643e98da94752f4de43b8 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:20.115747 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c868ae9dcad643e98da94752f4de43b8" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:20.115962 9343 raft_consensus.cc:2454] T c868ae9dcad643e98da94752f4de43b8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:20.116144 8883 leader_election.cc:302] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:20.116535 9414 raft_consensus.cc:2781] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:20.116595 9414 raft_consensus.cc:482] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:20.116631 9414 raft_consensus.cc:3037] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:20.117453 9414 raft_consensus.cc:504] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:20.117785 9414 leader_election.cc:288] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:20.118096 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c868ae9dcad643e98da94752f4de43b8" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:20.118206 9343 raft_consensus.cc:3037] T c868ae9dcad643e98da94752f4de43b8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:20.118274 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c868ae9dcad643e98da94752f4de43b8" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:20.118373 9081 raft_consensus.cc:3037] T c868ae9dcad643e98da94752f4de43b8 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:20.119179 9343 raft_consensus.cc:2454] T c868ae9dcad643e98da94752f4de43b8 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:20.119251 9081 raft_consensus.cc:2454] T c868ae9dcad643e98da94752f4de43b8 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:20.119529 8884 leader_election.cc:302] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 0d77752181e944698e809e325e5937e6; no voters:
I20240503 13:59:20.119731 9414 raft_consensus.cc:2781] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:20.119968 9414 raft_consensus.cc:686] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 13:59:20.120186 9414 consensus_queue.cc:237] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:20.122257 6379 catalog_manager.cc:5484] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:20.236107 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:20.240813 9343 raft_consensus.cc:1261] T c868ae9dcad643e98da94752f4de43b8 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:20.240869 9081 raft_consensus.cc:1261] T c868ae9dcad643e98da94752f4de43b8 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:20.241493 9621 consensus_queue.cc:1035] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:20.241892 9414 consensus_queue.cc:1035] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:20.244131 6379 catalog_manager.cc:5576] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931) does not have the latest schema for tablet c868ae9dcad643e98da94752f4de43b8 (table rebalancer_dynamic_table [id=1ced52dfa09a490fbf1d05b376fc12af]). Expected version 1 got 0
I20240503 13:59:20.244958 9480 tablet.cc:1722] T c868ae9dcad643e98da94752f4de43b8 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:20.245092 9475 mvcc.cc:205] Tried to move back new op lower bound from 7023594537939828736 to 7023594537454034944. Current Snapshot: MvccSnapshot[applied={T|T < 7023594537939828736}]
I20240503 13:59:20.245260 9480 tablet.cc:1620] T c868ae9dcad643e98da94752f4de43b8 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:20.245507 9482 tablet.cc:1722] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:20.245793 9482 tablet.cc:1620] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:20.246402 6385 catalog_manager.cc:5576] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481) does not have the latest schema for tablet c868ae9dcad643e98da94752f4de43b8 (table rebalancer_dynamic_table [id=1ced52dfa09a490fbf1d05b376fc12af]). Expected version 1 got 0
I20240503 13:59:20.249979 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=1ced52dfa09a490fbf1d05b376fc12af] alter complete (version 1)
I20240503 13:59:20.250344 9626 tablet.cc:1722] T c868ae9dcad643e98da94752f4de43b8 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:20.251020 6379 catalog_manager.cc:5576] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481) does not have the latest schema for tablet c868ae9dcad643e98da94752f4de43b8 (table rebalancer_dynamic_table [id=1ced52dfa09a490fbf1d05b376fc12af]). Expected version 1 got 0
I20240503 13:59:20.251176 9626 tablet.cc:1620] T c868ae9dcad643e98da94752f4de43b8 P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:20.296249 6379 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240503 13:59:20.296407 6379 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240503 13:59:20.296584 8743 consensus_queue.cc:1035] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:20.299407 6379 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 12d82a64f2f543f582d154ac47c0f0cb
I20240503 13:59:20.299722 6379 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 94eeec82221e48f6918a2f56e7aa5f37
I20240503 13:59:20.299966 6379 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 0e889726631646499b8c11742ee0185f
I20240503 13:59:20.300065 6640 tablet_service.cc:1514] Processing DeleteTablet for tablet 12d82a64f2f543f582d154ac47c0f0cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:20 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:20.300128 8743 consensus_queue.cc:1035] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:20.300248 9540 tablet_replica.cc:330] T 12d82a64f2f543f582d154ac47c0f0cb P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:20.300375 9540 raft_consensus.cc:2229] T 12d82a64f2f543f582d154ac47c0f0cb P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:20.300518 9540 raft_consensus.cc:2258] T 12d82a64f2f543f582d154ac47c0f0cb P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:20.300606 6771 tablet_service.cc:1514] Processing DeleteTablet for tablet 94eeec82221e48f6918a2f56e7aa5f37 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:20 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:20.301079 9540 ts_tablet_manager.cc:1875] T 12d82a64f2f543f582d154ac47c0f0cb P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:20.301123 9542 tablet_replica.cc:330] T 94eeec82221e48f6918a2f56e7aa5f37 P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:20.301103 6509 tablet_service.cc:1514] Processing DeleteTablet for tablet 12d82a64f2f543f582d154ac47c0f0cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:20 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:20.301254 9542 raft_consensus.cc:2229] T 94eeec82221e48f6918a2f56e7aa5f37 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:20.301409 9542 raft_consensus.cc:2258] T 94eeec82221e48f6918a2f56e7aa5f37 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:20.301574 9541 tablet_replica.cc:330] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:20.301710 9541 raft_consensus.cc:2229] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:20.301911 9542 ts_tablet_manager.cc:1875] T 94eeec82221e48f6918a2f56e7aa5f37 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:20.301945 9541 raft_consensus.cc:2258] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:20.302448 9541 ts_tablet_manager.cc:1875] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:20.303246 6507 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e889726631646499b8c11742ee0185f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:20 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:20.303372 6508 tablet_service.cc:1514] Processing DeleteTablet for tablet 94eeec82221e48f6918a2f56e7aa5f37 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:20 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:20.303436 9540 ts_tablet_manager.cc:1888] T 12d82a64f2f543f582d154ac47c0f0cb P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:20.303499 9540 log.cc:1196] T 12d82a64f2f543f582d154ac47c0f0cb P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/12d82a64f2f543f582d154ac47c0f0cb
I20240503 13:59:20.303778 9540 ts_tablet_manager.cc:1909] T 12d82a64f2f543f582d154ac47c0f0cb P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:20.304450 9541 ts_tablet_manager.cc:1888] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:20.304499 6900 tablet_service.cc:1514] Processing DeleteTablet for tablet 12d82a64f2f543f582d154ac47c0f0cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:20 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:20.304584 6901 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e889726631646499b8c11742ee0185f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:20 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:20.304677 9543 tablet_replica.cc:330] T 12d82a64f2f543f582d154ac47c0f0cb P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:20.304801 9543 raft_consensus.cc:2229] T 12d82a64f2f543f582d154ac47c0f0cb P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:20.304836 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet 12d82a64f2f543f582d154ac47c0f0cb (table rebalance_test_table_2 [id=fb49f602527d4645be66ec1c72a6702d]) successfully deleted
I20240503 13:59:20.304836 6640 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e889726631646499b8c11742ee0185f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:20 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:20.304926 9543 raft_consensus.cc:2258] T 12d82a64f2f543f582d154ac47c0f0cb P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:20.305002 9540 tablet_replica.cc:330] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:20.305068 6902 tablet_service.cc:1514] Processing DeleteTablet for tablet 94eeec82221e48f6918a2f56e7aa5f37 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:20 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:20.305114 9540 raft_consensus.cc:2229] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:20.305371 9540 raft_consensus.cc:2258] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:20.305405 9542 ts_tablet_manager.cc:1888] T 94eeec82221e48f6918a2f56e7aa5f37 P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:20.305424 9543 ts_tablet_manager.cc:1875] T 12d82a64f2f543f582d154ac47c0f0cb P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:20.305464 9542 log.cc:1196] T 94eeec82221e48f6918a2f56e7aa5f37 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/94eeec82221e48f6918a2f56e7aa5f37
I20240503 13:59:20.305717 9542 ts_tablet_manager.cc:1909] T 94eeec82221e48f6918a2f56e7aa5f37 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:20.304535 9541 log.cc:1196] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/12d82a64f2f543f582d154ac47c0f0cb
I20240503 13:59:20.305918 9540 ts_tablet_manager.cc:1875] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:20.306200 9541 ts_tablet_manager.cc:1909] T 12d82a64f2f543f582d154ac47c0f0cb P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:20.306360 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet 94eeec82221e48f6918a2f56e7aa5f37 (table rebalance_test_table_2 [id=fb49f602527d4645be66ec1c72a6702d]) successfully deleted
I20240503 13:59:20.306933 9541 tablet_replica.cc:330] T 0e889726631646499b8c11742ee0185f P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:20.307111 9541 raft_consensus.cc:2229] T 0e889726631646499b8c11742ee0185f P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:20.307276 9541 raft_consensus.cc:2258] T 0e889726631646499b8c11742ee0185f P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:20.307612 9543 ts_tablet_manager.cc:1888] T 12d82a64f2f543f582d154ac47c0f0cb P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:20.307685 9543 log.cc:1196] T 12d82a64f2f543f582d154ac47c0f0cb P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/12d82a64f2f543f582d154ac47c0f0cb
I20240503 13:59:20.307854 9541 ts_tablet_manager.cc:1875] T 0e889726631646499b8c11742ee0185f P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:20.307933 9543 ts_tablet_manager.cc:1909] T 12d82a64f2f543f582d154ac47c0f0cb P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:20.308174 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 12d82a64f2f543f582d154ac47c0f0cb (table rebalance_test_table_2 [id=fb49f602527d4645be66ec1c72a6702d]) successfully deleted
I20240503 13:59:20.308214 9540 ts_tablet_manager.cc:1888] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:20.308274 9540 log.cc:1196] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/0e889726631646499b8c11742ee0185f
I20240503 13:59:20.308537 9540 ts_tablet_manager.cc:1909] T 0e889726631646499b8c11742ee0185f P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:20.308666 9543 tablet_replica.cc:330] T 0e889726631646499b8c11742ee0185f P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:20.309238 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet 0e889726631646499b8c11742ee0185f (table rebalance_test_table_2 [id=fb49f602527d4645be66ec1c72a6702d]) successfully deleted
I20240503 13:59:20.309662 9541 ts_tablet_manager.cc:1888] T 0e889726631646499b8c11742ee0185f P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:20.309723 9541 log.cc:1196] T 0e889726631646499b8c11742ee0185f P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0e889726631646499b8c11742ee0185f
I20240503 13:59:20.309932 9541 ts_tablet_manager.cc:1909] T 0e889726631646499b8c11742ee0185f P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:20.310091 9543 raft_consensus.cc:2229] T 0e889726631646499b8c11742ee0185f P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:20.310127 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet 12d82a64f2f543f582d154ac47c0f0cb (table rebalance_test_table_2 [id=fb49f602527d4645be66ec1c72a6702d]) successfully deleted
I20240503 13:59:20.310249 9543 raft_consensus.cc:2258] T 0e889726631646499b8c11742ee0185f P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:20.310601 9541 tablet_replica.cc:330] T 94eeec82221e48f6918a2f56e7aa5f37 P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:20.310695 9543 ts_tablet_manager.cc:1875] T 0e889726631646499b8c11742ee0185f P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:20.310721 9541 raft_consensus.cc:2229] T 94eeec82221e48f6918a2f56e7aa5f37 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:20.310711 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 0e889726631646499b8c11742ee0185f (table rebalance_test_table_2 [id=fb49f602527d4645be66ec1c72a6702d]) successfully deleted
I20240503 13:59:20.310833 9541 raft_consensus.cc:2258] T 94eeec82221e48f6918a2f56e7aa5f37 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:20.311316 9541 ts_tablet_manager.cc:1875] T 94eeec82221e48f6918a2f56e7aa5f37 P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:20.312435 9543 ts_tablet_manager.cc:1888] T 0e889726631646499b8c11742ee0185f P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:20.312508 9543 log.cc:1196] T 0e889726631646499b8c11742ee0185f P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/0e889726631646499b8c11742ee0185f
I20240503 13:59:20.312755 9543 ts_tablet_manager.cc:1909] T 0e889726631646499b8c11742ee0185f P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:20.312762 9541 ts_tablet_manager.cc:1888] T 94eeec82221e48f6918a2f56e7aa5f37 P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:20.312826 9541 log.cc:1196] T 94eeec82221e48f6918a2f56e7aa5f37 P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/94eeec82221e48f6918a2f56e7aa5f37
I20240503 13:59:20.313120 9541 ts_tablet_manager.cc:1909] T 94eeec82221e48f6918a2f56e7aa5f37 P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:20.313251 9543 tablet_replica.cc:330] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:20.313367 9543 raft_consensus.cc:2229] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:20.313380 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet 0e889726631646499b8c11742ee0185f (table rebalance_test_table_2 [id=fb49f602527d4645be66ec1c72a6702d]) successfully deleted
I20240503 13:59:20.313550 9543 raft_consensus.cc:2258] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:20.313776 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 94eeec82221e48f6918a2f56e7aa5f37 (table rebalance_test_table_2 [id=fb49f602527d4645be66ec1c72a6702d]) successfully deleted
I20240503 13:59:20.314013 9543 ts_tablet_manager.cc:1875] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:20.315371 9543 ts_tablet_manager.cc:1888] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:20.315430 9543 log.cc:1196] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/94eeec82221e48f6918a2f56e7aa5f37
I20240503 13:59:20.315644 9543 ts_tablet_manager.cc:1909] T 94eeec82221e48f6918a2f56e7aa5f37 P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:20.316169 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet 94eeec82221e48f6918a2f56e7aa5f37 (table rebalance_test_table_2 [id=fb49f602527d4645be66ec1c72a6702d]) successfully deleted
I20240503 13:59:20.353402 6385 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:20.358968 9482 tablet.cc:1722] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:20.359659 9482 tablet.cc:1620] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:20.359822 9480 tablet.cc:1722] T c868ae9dcad643e98da94752f4de43b8 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:20.360121 9480 tablet.cc:1620] T c868ae9dcad643e98da94752f4de43b8 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:20.360361 9626 tablet.cc:1722] T c868ae9dcad643e98da94752f4de43b8 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:20.360587 9626 tablet.cc:1620] T c868ae9dcad643e98da94752f4de43b8 P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:20.364236 6385 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=1ced52dfa09a490fbf1d05b376fc12af] alter complete (version 2)
I20240503 13:59:20.466029 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:20.467783 8600 consensus_queue.cc:1035] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:20.471076 8758 consensus_queue.cc:1035] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:20.472875 9482 tablet.cc:1722] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:20.474144 9626 tablet.cc:1722] T c868ae9dcad643e98da94752f4de43b8 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:20.474898 9480 tablet.cc:1722] T c868ae9dcad643e98da94752f4de43b8 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:20.479535 6385 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=1ced52dfa09a490fbf1d05b376fc12af] alter complete (version 3)
I20240503 13:59:20.548961 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:20.554519 9192 tablet_service.cc:1467] Processing CreateTablet for tablet 564582aa33a7465081724523096591d3 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=e92a6218e0734bbc97220157e127bc15]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:20.554596 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 564582aa33a7465081724523096591d3 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=e92a6218e0734bbc97220157e127bc15]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:20.554858 9192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 564582aa33a7465081724523096591d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:20.554908 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 564582aa33a7465081724523096591d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:20.555333 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 1e5875904e6047eb82f28c3cbc9318fc (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=e92a6218e0734bbc97220157e127bc15]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:20.555428 9323 tablet_service.cc:1467] Processing CreateTablet for tablet 1e5875904e6047eb82f28c3cbc9318fc (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=e92a6218e0734bbc97220157e127bc15]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:20.555585 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e5875904e6047eb82f28c3cbc9318fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:20.555774 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e5875904e6047eb82f28c3cbc9318fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:20.557372 9192 tablet_service.cc:1467] Processing CreateTablet for tablet 1e5875904e6047eb82f28c3cbc9318fc (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=e92a6218e0734bbc97220157e127bc15]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:20.557567 9412 tablet_bootstrap.cc:492] T 1e5875904e6047eb82f28c3cbc9318fc P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:20.557622 9192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e5875904e6047eb82f28c3cbc9318fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:20.557627 9552 tablet_bootstrap.cc:492] T 564582aa33a7465081724523096591d3 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:20.558470 9412 tablet_bootstrap.cc:654] T 1e5875904e6047eb82f28c3cbc9318fc P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:20.558957 9322 tablet_service.cc:1467] Processing CreateTablet for tablet bccac59f680046389bbe23ff262a82d6 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=e92a6218e0734bbc97220157e127bc15]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:20.559202 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bccac59f680046389bbe23ff262a82d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:20.559614 9412 tablet_bootstrap.cc:492] T 1e5875904e6047eb82f28c3cbc9318fc P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:20.559693 9412 ts_tablet_manager.cc:1367] T 1e5875904e6047eb82f28c3cbc9318fc P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:20.559855 9552 tablet_bootstrap.cc:654] T 564582aa33a7465081724523096591d3 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:20.559866 9634 tablet_bootstrap.cc:492] T 564582aa33a7465081724523096591d3 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:20.560155 9412 raft_consensus.cc:348] T 1e5875904e6047eb82f28c3cbc9318fc P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:20.560250 9412 raft_consensus.cc:374] T 1e5875904e6047eb82f28c3cbc9318fc P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:20.560281 9412 raft_consensus.cc:729] T 1e5875904e6047eb82f28c3cbc9318fc P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:20.560410 9412 consensus_queue.cc:260] T 1e5875904e6047eb82f28c3cbc9318fc P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:20.560657 9412 ts_tablet_manager.cc:1398] T 1e5875904e6047eb82f28c3cbc9318fc P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:20.561384 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 564582aa33a7465081724523096591d3 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=e92a6218e0734bbc97220157e127bc15]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:20.561650 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 564582aa33a7465081724523096591d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:20.562285 9552 tablet_bootstrap.cc:492] T 564582aa33a7465081724523096591d3 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:20.562361 9634 tablet_bootstrap.cc:654] T 564582aa33a7465081724523096591d3 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:20.562374 9552 ts_tablet_manager.cc:1367] T 564582aa33a7465081724523096591d3 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:20.562728 9552 raft_consensus.cc:348] T 564582aa33a7465081724523096591d3 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:20.562813 9552 raft_consensus.cc:374] T 564582aa33a7465081724523096591d3 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:20.562844 9552 raft_consensus.cc:729] T 564582aa33a7465081724523096591d3 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:20.562959 9552 consensus_queue.cc:260] T 564582aa33a7465081724523096591d3 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:20.563195 9552 ts_tablet_manager.cc:1398] T 564582aa33a7465081724523096591d3 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:20.564188 9634 tablet_bootstrap.cc:492] T 564582aa33a7465081724523096591d3 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:20.564330 9639 tablet_bootstrap.cc:492] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:20.564610 9634 ts_tablet_manager.cc:1367] T 564582aa33a7465081724523096591d3 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:20.564886 9058 tablet_service.cc:1467] Processing CreateTablet for tablet bccac59f680046389bbe23ff262a82d6 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=e92a6218e0734bbc97220157e127bc15]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:20.565128 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bccac59f680046389bbe23ff262a82d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:20.565251 9634 raft_consensus.cc:348] T 564582aa33a7465081724523096591d3 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:20.565352 9634 raft_consensus.cc:374] T 564582aa33a7465081724523096591d3 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:20.565387 9634 raft_consensus.cc:729] T 564582aa33a7465081724523096591d3 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:20.565737 9634 consensus_queue.cc:260] T 564582aa33a7465081724523096591d3 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:20.566040 9639 tablet_bootstrap.cc:654] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:20.566895 9634 ts_tablet_manager.cc:1398] T 564582aa33a7465081724523096591d3 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:20.567071 9634 tablet_bootstrap.cc:492] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:20.569097 9634 tablet_bootstrap.cc:654] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:20.570298 9639 tablet_bootstrap.cc:492] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:20.571270 9639 ts_tablet_manager.cc:1367] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:59:20.571396 9634 tablet_bootstrap.cc:492] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:20.571475 9634 ts_tablet_manager.cc:1367] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:20.571857 9634 raft_consensus.cc:348] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:20.571969 9634 raft_consensus.cc:374] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:20.572001 9634 raft_consensus.cc:729] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:20.572116 9634 consensus_queue.cc:260] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:20.571857 9639 raft_consensus.cc:348] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:20.572331 9639 raft_consensus.cc:374] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:20.572407 9639 raft_consensus.cc:729] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:20.572461 9634 ts_tablet_manager.cc:1398] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:20.572628 9639 consensus_queue.cc:260] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:20.573024 9639 ts_tablet_manager.cc:1398] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:20.573444 9610 consensus_queue.cc:1035] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:20.577067 9552 tablet_bootstrap.cc:492] T bccac59f680046389bbe23ff262a82d6 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:20.578110 9552 tablet_bootstrap.cc:654] T bccac59f680046389bbe23ff262a82d6 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:20.579898 9643 consensus_queue.cc:1035] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:20.580879 8928 tablet_service.cc:1467] Processing CreateTablet for tablet bccac59f680046389bbe23ff262a82d6 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=e92a6218e0734bbc97220157e127bc15]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:20.581169 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bccac59f680046389bbe23ff262a82d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:20.583753 9412 tablet_bootstrap.cc:492] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:20.584332 6380 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:20.584522 6380 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:20.585954 9412 tablet_bootstrap.cc:654] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:20.586133 9552 tablet_bootstrap.cc:492] T bccac59f680046389bbe23ff262a82d6 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:20.586217 9552 ts_tablet_manager.cc:1367] T bccac59f680046389bbe23ff262a82d6 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240503 13:59:20.586654 9552 raft_consensus.cc:348] T bccac59f680046389bbe23ff262a82d6 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:20.586758 9552 raft_consensus.cc:374] T bccac59f680046389bbe23ff262a82d6 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:20.586789 9552 raft_consensus.cc:729] T bccac59f680046389bbe23ff262a82d6 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:20.586911 9552 consensus_queue.cc:260] T bccac59f680046389bbe23ff262a82d6 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:20.587186 9552 ts_tablet_manager.cc:1398] T bccac59f680046389bbe23ff262a82d6 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:20.587360 6380 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet c868ae9dcad643e98da94752f4de43b8
I20240503 13:59:20.588196 8928 tablet_service.cc:1514] Processing DeleteTablet for tablet c868ae9dcad643e98da94752f4de43b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:20 UTC) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:20.588850 9058 tablet_service.cc:1514] Processing DeleteTablet for tablet c868ae9dcad643e98da94752f4de43b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:20 UTC) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:20.589310 9649 tablet_replica.cc:330] T c868ae9dcad643e98da94752f4de43b8 P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:20.589373 9323 tablet_service.cc:1514] Processing DeleteTablet for tablet c868ae9dcad643e98da94752f4de43b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:20 UTC) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:20.589673 9650 tablet_replica.cc:330] T c868ae9dcad643e98da94752f4de43b8 P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:20.589663 9649 raft_consensus.cc:2229] T c868ae9dcad643e98da94752f4de43b8 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:20.589780 9650 raft_consensus.cc:2229] T c868ae9dcad643e98da94752f4de43b8 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:20.589879 9649 raft_consensus.cc:2258] T c868ae9dcad643e98da94752f4de43b8 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:20.590036 9650 raft_consensus.cc:2258] T c868ae9dcad643e98da94752f4de43b8 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:20.590152 9637 tablet_bootstrap.cc:492] T bccac59f680046389bbe23ff262a82d6 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:20.590606 9650 ts_tablet_manager.cc:1875] T c868ae9dcad643e98da94752f4de43b8 P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:20.591104 9649 ts_tablet_manager.cc:1875] T c868ae9dcad643e98da94752f4de43b8 P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:20.591589 9412 tablet_bootstrap.cc:492] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:20.591686 9412 ts_tablet_manager.cc:1367] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240503 13:59:20.592149 9412 raft_consensus.cc:348] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:20.592263 9412 raft_consensus.cc:374] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:20.592300 9412 raft_consensus.cc:729] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:20.592453 9637 tablet_bootstrap.cc:654] T bccac59f680046389bbe23ff262a82d6 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:20.592701 9412 consensus_queue.cc:260] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:20.593549 9412 ts_tablet_manager.cc:1398] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:20.593616 9650 ts_tablet_manager.cc:1888] T c868ae9dcad643e98da94752f4de43b8 P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:20.593684 9650 log.cc:1196] T c868ae9dcad643e98da94752f4de43b8 P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/c868ae9dcad643e98da94752f4de43b8
I20240503 13:59:20.593845 9551 raft_consensus.cc:482] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:20.593955 9551 raft_consensus.cc:504] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:20.594043 9650 ts_tablet_manager.cc:1909] T c868ae9dcad643e98da94752f4de43b8 P 0d77752181e944698e809e325e5937e6: Deleting consensus metadata
I20240503 13:59:20.594444 9648 tablet_replica.cc:330] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:20.594522 9551 leader_election.cc:288] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:20.594560 9648 raft_consensus.cc:2229] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:20.594842 9648 raft_consensus.cc:2258] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:20.595528 9649 ts_tablet_manager.cc:1888] T c868ae9dcad643e98da94752f4de43b8 P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:20.595623 9649 log.cc:1196] T c868ae9dcad643e98da94752f4de43b8 P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/c868ae9dcad643e98da94752f4de43b8
I20240503 13:59:20.596312 9649 ts_tablet_manager.cc:1909] T c868ae9dcad643e98da94752f4de43b8 P 8d574fb4a89d4eea947c859470d44dd4: Deleting consensus metadata
I20240503 13:59:20.596621 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet c868ae9dcad643e98da94752f4de43b8 (table rebalancer_dynamic_table_new_name [id=1ced52dfa09a490fbf1d05b376fc12af]) successfully deleted
I20240503 13:59:20.596906 9648 ts_tablet_manager.cc:1875] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:20.597751 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet c868ae9dcad643e98da94752f4de43b8 (table rebalancer_dynamic_table_new_name [id=1ced52dfa09a490fbf1d05b376fc12af]) successfully deleted
I20240503 13:59:20.599025 9637 tablet_bootstrap.cc:492] T bccac59f680046389bbe23ff262a82d6 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:20.599161 9637 ts_tablet_manager.cc:1367] T bccac59f680046389bbe23ff262a82d6 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:59:20.599495 9648 ts_tablet_manager.cc:1888] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:20.599571 9648 log.cc:1196] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/c868ae9dcad643e98da94752f4de43b8
I20240503 13:59:20.599825 9648 ts_tablet_manager.cc:1909] T c868ae9dcad643e98da94752f4de43b8 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting consensus metadata
I20240503 13:59:20.601343 9637 raft_consensus.cc:348] T bccac59f680046389bbe23ff262a82d6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:20.601509 9637 raft_consensus.cc:374] T bccac59f680046389bbe23ff262a82d6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:20.601581 9637 raft_consensus.cc:729] T bccac59f680046389bbe23ff262a82d6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:20.601652 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564582aa33a7465081724523096591d3" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:20.601735 9637 consensus_queue.cc:260] T bccac59f680046389bbe23ff262a82d6 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:20.601795 9080 raft_consensus.cc:2454] T 564582aa33a7465081724523096591d3 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:20.601974 9637 ts_tablet_manager.cc:1398] T bccac59f680046389bbe23ff262a82d6 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:20.602115 9637 tablet_bootstrap.cc:492] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:20.601431 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet c868ae9dcad643e98da94752f4de43b8 (table rebalancer_dynamic_table_new_name [id=1ced52dfa09a490fbf1d05b376fc12af]) successfully deleted
I20240503 13:59:20.602746 6595 leader_election.cc:302] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:20.603075 9637 tablet_bootstrap.cc:654] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:20.603722 9551 raft_consensus.cc:2781] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:20.603792 9551 raft_consensus.cc:482] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:20.603830 9551 raft_consensus.cc:3037] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:20.604867 9551 raft_consensus.cc:504] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:20.605243 9551 leader_election.cc:288] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:20.605648 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564582aa33a7465081724523096591d3" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:20.605772 9080 raft_consensus.cc:3037] T 564582aa33a7465081724523096591d3 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:20.606566 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564582aa33a7465081724523096591d3" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:20.606693 9211 raft_consensus.cc:2454] T 564582aa33a7465081724523096591d3 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:20.606838 9080 raft_consensus.cc:2454] T 564582aa33a7465081724523096591d3 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:20.606925 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564582aa33a7465081724523096591d3" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:20.607021 9205 raft_consensus.cc:3037] T 564582aa33a7465081724523096591d3 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:20.607221 6595 leader_election.cc:302] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:20.607920 9205 raft_consensus.cc:2454] T 564582aa33a7465081724523096591d3 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:20.608094 9551 raft_consensus.cc:2781] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:20.608198 9551 raft_consensus.cc:686] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 13:59:20.608340 9551 consensus_queue.cc:237] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:20.609776 9637 tablet_bootstrap.cc:492] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:20.609936 9637 ts_tablet_manager.cc:1367] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240503 13:59:20.610491 6380 catalog_manager.cc:5484] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:20.610592 9637 raft_consensus.cc:348] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:20.610679 9637 raft_consensus.cc:374] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:20.610754 9637 raft_consensus.cc:729] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:20.610929 9637 consensus_queue.cc:260] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:20.611209 9637 ts_tablet_manager.cc:1398] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:20.640290 9408 raft_consensus.cc:482] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:20.640419 9408 raft_consensus.cc:504] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:20.640852 9408 leader_election.cc:288] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:20.641335 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e5875904e6047eb82f28c3cbc9318fc" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:20.641364 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e5875904e6047eb82f28c3cbc9318fc" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:20.641481 8948 raft_consensus.cc:2454] T 1e5875904e6047eb82f28c3cbc9318fc P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:20.641481 9205 raft_consensus.cc:2454] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:20.641795 9278 leader_election.cc:302] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 0d77752181e944698e809e325e5937e6; no voters:
I20240503 13:59:20.642057 9408 raft_consensus.cc:2781] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:20.642112 9408 raft_consensus.cc:482] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:20.642164 9408 raft_consensus.cc:3037] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:20.643183 9408 raft_consensus.cc:504] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:20.643529 9408 leader_election.cc:288] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:20.643889 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e5875904e6047eb82f28c3cbc9318fc" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:20.643901 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e5875904e6047eb82f28c3cbc9318fc" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:20.644002 9205 raft_consensus.cc:3037] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:20.644037 8948 raft_consensus.cc:3037] T 1e5875904e6047eb82f28c3cbc9318fc P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:20.644894 9205 raft_consensus.cc:2454] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:20.644935 8948 raft_consensus.cc:2454] T 1e5875904e6047eb82f28c3cbc9318fc P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:20.645152 9278 leader_election.cc:302] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:20.645306 9408 raft_consensus.cc:2781] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:20.645462 9408 raft_consensus.cc:686] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:20.645618 9408 consensus_queue.cc:237] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:20.647089 6380 catalog_manager.cc:5484] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:20.690464 6380 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:20.695695 9058 tablet_service.cc:1467] Processing CreateTablet for tablet e572f8c125044029809ac8c69fa3bad8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=61aae4c8eda1440ba9af9c3cd9eec9a7]), partition=
I20240503 13:59:20.695981 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e572f8c125044029809ac8c69fa3bad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:20.695942 9323 tablet_service.cc:1467] Processing CreateTablet for tablet e572f8c125044029809ac8c69fa3bad8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=61aae4c8eda1440ba9af9c3cd9eec9a7]), partition=
I20240503 13:59:20.696173 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e572f8c125044029809ac8c69fa3bad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:20.698120 9552 tablet_bootstrap.cc:492] T e572f8c125044029809ac8c69fa3bad8 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:20.698120 9637 tablet_bootstrap.cc:492] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:20.698488 8928 tablet_service.cc:1467] Processing CreateTablet for tablet e572f8c125044029809ac8c69fa3bad8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=61aae4c8eda1440ba9af9c3cd9eec9a7]), partition=
I20240503 13:59:20.698709 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e572f8c125044029809ac8c69fa3bad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:20.699061 9637 tablet_bootstrap.cc:654] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:20.699061 9552 tablet_bootstrap.cc:654] T e572f8c125044029809ac8c69fa3bad8 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:20.700263 9552 tablet_bootstrap.cc:492] T e572f8c125044029809ac8c69fa3bad8 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:20.700263 9637 tablet_bootstrap.cc:492] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:20.700351 9552 ts_tablet_manager.cc:1367] T e572f8c125044029809ac8c69fa3bad8 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 13:59:20.700351 9637 ts_tablet_manager.cc:1367] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:20.700520 9412 tablet_bootstrap.cc:492] T e572f8c125044029809ac8c69fa3bad8 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:20.700805 9637 raft_consensus.cc:348] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:20.700803 9552 raft_consensus.cc:348] T e572f8c125044029809ac8c69fa3bad8 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:20.700912 9552 raft_consensus.cc:374] T e572f8c125044029809ac8c69fa3bad8 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:20.700912 9637 raft_consensus.cc:374] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:20.700953 9552 raft_consensus.cc:729] T e572f8c125044029809ac8c69fa3bad8 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:20.700953 9637 raft_consensus.cc:729] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:20.701066 9552 consensus_queue.cc:260] T e572f8c125044029809ac8c69fa3bad8 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:20.701066 9637 consensus_queue.cc:260] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:20.701249 9412 tablet_bootstrap.cc:654] T e572f8c125044029809ac8c69fa3bad8 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:20.701318 9637 ts_tablet_manager.cc:1398] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:20.701318 9552 ts_tablet_manager.cc:1398] T e572f8c125044029809ac8c69fa3bad8 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:20.703199 9412 tablet_bootstrap.cc:492] T e572f8c125044029809ac8c69fa3bad8 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:20.703306 9412 ts_tablet_manager.cc:1367] T e572f8c125044029809ac8c69fa3bad8 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:20.703711 9412 raft_consensus.cc:348] T e572f8c125044029809ac8c69fa3bad8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:20.703832 9412 raft_consensus.cc:374] T e572f8c125044029809ac8c69fa3bad8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:20.703881 9412 raft_consensus.cc:729] T e572f8c125044029809ac8c69fa3bad8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:20.704005 9412 consensus_queue.cc:260] T e572f8c125044029809ac8c69fa3bad8 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:20.704265 9412 ts_tablet_manager.cc:1398] T e572f8c125044029809ac8c69fa3bad8 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:20.720707 9414 raft_consensus.cc:482] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:20.720816 9414 raft_consensus.cc:504] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:20.721161 9414 leader_election.cc:288] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:20.721521 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bccac59f680046389bbe23ff262a82d6" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:20.721566 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bccac59f680046389bbe23ff262a82d6" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:20.721660 9080 raft_consensus.cc:2454] T bccac59f680046389bbe23ff262a82d6 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:20.721714 9343 raft_consensus.cc:2454] T bccac59f680046389bbe23ff262a82d6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:20.721952 8883 leader_election.cc:302] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:20.722153 9414 raft_consensus.cc:2781] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:20.722229 9414 raft_consensus.cc:482] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:20.722273 9414 raft_consensus.cc:3037] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:20.723093 9414 raft_consensus.cc:504] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:20.723393 9414 leader_election.cc:288] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:20.723687 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bccac59f680046389bbe23ff262a82d6" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:20.723711 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bccac59f680046389bbe23ff262a82d6" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:20.723783 9080 raft_consensus.cc:3037] T bccac59f680046389bbe23ff262a82d6 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:20.723812 9343 raft_consensus.cc:3037] T bccac59f680046389bbe23ff262a82d6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:20.724526 9080 raft_consensus.cc:2454] T bccac59f680046389bbe23ff262a82d6 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:20.724792 8883 leader_election.cc:302] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:20.724936 9414 raft_consensus.cc:2781] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:20.724948 9343 raft_consensus.cc:2454] T bccac59f680046389bbe23ff262a82d6 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:20.725013 9414 raft_consensus.cc:686] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 13:59:20.725159 9414 consensus_queue.cc:237] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:20.726673 6380 catalog_manager.cc:5484] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:20.804359 6380 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240503 13:59:20.804577 6380 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240503 13:59:20.806761 6380 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 712bb27624184da8ba9ba0d1248abc02
I20240503 13:59:20.807065 6380 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 63ec50dbf2624c24bc0e9325ab28bb86
I20240503 13:59:20.807304 6508 tablet_service.cc:1514] Processing DeleteTablet for tablet 712bb27624184da8ba9ba0d1248abc02 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:20 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:20.807310 6640 tablet_service.cc:1514] Processing DeleteTablet for tablet 712bb27624184da8ba9ba0d1248abc02 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:20 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:20.807423 6771 tablet_service.cc:1514] Processing DeleteTablet for tablet 712bb27624184da8ba9ba0d1248abc02 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:20 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:20.807492 9540 tablet_replica.cc:330] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:20.807621 9542 tablet_replica.cc:330] T 712bb27624184da8ba9ba0d1248abc02 P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:20.807629 9541 tablet_replica.cc:330] T 712bb27624184da8ba9ba0d1248abc02 P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:20.807648 9540 raft_consensus.cc:2229] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:20.807746 9542 raft_consensus.cc:2229] T 712bb27624184da8ba9ba0d1248abc02 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:20.807760 9541 raft_consensus.cc:2229] T 712bb27624184da8ba9ba0d1248abc02 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:20.807832 9540 raft_consensus.cc:2258] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:20.807888 9542 raft_consensus.cc:2258] T 712bb27624184da8ba9ba0d1248abc02 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:20.807890 9541 raft_consensus.cc:2258] T 712bb27624184da8ba9ba0d1248abc02 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:20.808328 9541 ts_tablet_manager.cc:1875] T 712bb27624184da8ba9ba0d1248abc02 P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:20.808358 9540 ts_tablet_manager.cc:1875] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:20.808358 9542 ts_tablet_manager.cc:1875] T 712bb27624184da8ba9ba0d1248abc02 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:20.810716 9541 ts_tablet_manager.cc:1888] T 712bb27624184da8ba9ba0d1248abc02 P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:20.810789 9542 ts_tablet_manager.cc:1888] T 712bb27624184da8ba9ba0d1248abc02 P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:20.810846 9542 log.cc:1196] T 712bb27624184da8ba9ba0d1248abc02 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/712bb27624184da8ba9ba0d1248abc02
I20240503 13:59:20.810860 9541 log.cc:1196] T 712bb27624184da8ba9ba0d1248abc02 P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/712bb27624184da8ba9ba0d1248abc02
I20240503 13:59:20.811141 9542 ts_tablet_manager.cc:1909] T 712bb27624184da8ba9ba0d1248abc02 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:20.811141 9541 ts_tablet_manager.cc:1909] T 712bb27624184da8ba9ba0d1248abc02 P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:20.811218 6902 tablet_service.cc:1514] Processing DeleteTablet for tablet 63ec50dbf2624c24bc0e9325ab28bb86 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:20 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:20.811430 9543 tablet_replica.cc:330] T 63ec50dbf2624c24bc0e9325ab28bb86 P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:20.811597 9543 raft_consensus.cc:2229] T 63ec50dbf2624c24bc0e9325ab28bb86 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:20.811739 9543 raft_consensus.cc:2258] T 63ec50dbf2624c24bc0e9325ab28bb86 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:20.811784 6508 tablet_service.cc:1514] Processing DeleteTablet for tablet 63ec50dbf2624c24bc0e9325ab28bb86 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:20 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:20.811961 6380 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet ae76780e32bb4e5cb4e1c86403544173
I20240503 13:59:20.812239 9543 ts_tablet_manager.cc:1875] T 63ec50dbf2624c24bc0e9325ab28bb86 P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:20.812407 6902 tablet_service.cc:1514] Processing DeleteTablet for tablet ae76780e32bb4e5cb4e1c86403544173 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:20 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:20.813680 6508 tablet_service.cc:1514] Processing DeleteTablet for tablet ae76780e32bb4e5cb4e1c86403544173 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:20 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:20.814221 9541 tablet_replica.cc:330] T 63ec50dbf2624c24bc0e9325ab28bb86 P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:20.814272 9543 ts_tablet_manager.cc:1888] T 63ec50dbf2624c24bc0e9325ab28bb86 P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:20.814330 9543 log.cc:1196] T 63ec50dbf2624c24bc0e9325ab28bb86 P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/63ec50dbf2624c24bc0e9325ab28bb86
I20240503 13:59:20.814314 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 712bb27624184da8ba9ba0d1248abc02 (table rebalance_test_table_4 [id=1b29a48257a84332b723b8a0a9ce625b]) successfully deleted
I20240503 13:59:20.814359 9541 raft_consensus.cc:2229] T 63ec50dbf2624c24bc0e9325ab28bb86 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:20.814507 9541 raft_consensus.cc:2258] T 63ec50dbf2624c24bc0e9325ab28bb86 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:20.814597 9543 ts_tablet_manager.cc:1909] T 63ec50dbf2624c24bc0e9325ab28bb86 P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:20.814848 6640 tablet_service.cc:1514] Processing DeleteTablet for tablet 63ec50dbf2624c24bc0e9325ab28bb86 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:20 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:20.815016 9541 ts_tablet_manager.cc:1875] T 63ec50dbf2624c24bc0e9325ab28bb86 P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:20.815075 9540 ts_tablet_manager.cc:1888] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:20.815132 9540 log.cc:1196] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/712bb27624184da8ba9ba0d1248abc02
I20240503 13:59:20.815403 9540 ts_tablet_manager.cc:1909] T 712bb27624184da8ba9ba0d1248abc02 P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:20.815621 6771 tablet_service.cc:1514] Processing DeleteTablet for tablet ae76780e32bb4e5cb4e1c86403544173 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:20 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:20.815762 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet 63ec50dbf2624c24bc0e9325ab28bb86 (table rebalance_test_table_4 [id=1b29a48257a84332b723b8a0a9ce625b]) successfully deleted
I20240503 13:59:20.815933 9540 tablet_replica.cc:330] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:20.815965 9543 tablet_replica.cc:330] T ae76780e32bb4e5cb4e1c86403544173 P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:20.816051 9540 raft_consensus.cc:2229] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:20.816069 9543 raft_consensus.cc:2229] T ae76780e32bb4e5cb4e1c86403544173 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:20.816166 9543 raft_consensus.cc:2258] T ae76780e32bb4e5cb4e1c86403544173 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:20.816231 9540 raft_consensus.cc:2258] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:20.816586 9543 ts_tablet_manager.cc:1875] T ae76780e32bb4e5cb4e1c86403544173 P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:20.816711 9540 ts_tablet_manager.cc:1875] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:20.817734 9541 ts_tablet_manager.cc:1888] T 63ec50dbf2624c24bc0e9325ab28bb86 P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:20.817795 9541 log.cc:1196] T 63ec50dbf2624c24bc0e9325ab28bb86 P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/63ec50dbf2624c24bc0e9325ab28bb86
I20240503 13:59:20.818549 9541 ts_tablet_manager.cc:1909] T 63ec50dbf2624c24bc0e9325ab28bb86 P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:20.818682 9540 ts_tablet_manager.cc:1888] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:20.818744 9540 log.cc:1196] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/63ec50dbf2624c24bc0e9325ab28bb86
I20240503 13:59:20.818872 9543 ts_tablet_manager.cc:1888] T ae76780e32bb4e5cb4e1c86403544173 P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:20.818933 9543 log.cc:1196] T ae76780e32bb4e5cb4e1c86403544173 P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/ae76780e32bb4e5cb4e1c86403544173
I20240503 13:59:20.819033 9540 ts_tablet_manager.cc:1909] T 63ec50dbf2624c24bc0e9325ab28bb86 P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:20.819176 9543 ts_tablet_manager.cc:1909] T ae76780e32bb4e5cb4e1c86403544173 P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:20.819414 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet 712bb27624184da8ba9ba0d1248abc02 (table rebalance_test_table_4 [id=1b29a48257a84332b723b8a0a9ce625b]) successfully deleted
I20240503 13:59:20.819726 9541 tablet_replica.cc:330] T ae76780e32bb4e5cb4e1c86403544173 P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:20.819748 9408 raft_consensus.cc:482] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:20.819880 9542 tablet_replica.cc:330] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:20.819903 9541 raft_consensus.cc:2229] T ae76780e32bb4e5cb4e1c86403544173 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:20.819860 9408 raft_consensus.cc:504] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:20.819922 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 63ec50dbf2624c24bc0e9325ab28bb86 (table rebalance_test_table_4 [id=1b29a48257a84332b723b8a0a9ce625b]) successfully deleted
I20240503 13:59:20.819993 9542 raft_consensus.cc:2229] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:20.820027 9541 raft_consensus.cc:2258] T ae76780e32bb4e5cb4e1c86403544173 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:20.820214 9542 raft_consensus.cc:2258] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:20.820354 9408 leader_election.cc:288] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:20.820490 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet 712bb27624184da8ba9ba0d1248abc02 (table rebalance_test_table_4 [id=1b29a48257a84332b723b8a0a9ce625b]) successfully deleted
I20240503 13:59:20.820501 9541 ts_tablet_manager.cc:1875] T ae76780e32bb4e5cb4e1c86403544173 P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:20.820650 9542 ts_tablet_manager.cc:1875] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:20.821173 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet ae76780e32bb4e5cb4e1c86403544173 (table rebalance_test_table_4 [id=1b29a48257a84332b723b8a0a9ce625b]) successfully deleted
I20240503 13:59:20.821344 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e572f8c125044029809ac8c69fa3bad8" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:20.821486 8948 raft_consensus.cc:2454] T e572f8c125044029809ac8c69fa3bad8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:20.821868 9278 leader_election.cc:302] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 0d77752181e944698e809e325e5937e6; no voters:
I20240503 13:59:20.822049 9408 raft_consensus.cc:2781] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:20.822110 9408 raft_consensus.cc:482] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:20.822150 9408 raft_consensus.cc:3037] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:20.822468 9541 ts_tablet_manager.cc:1888] T ae76780e32bb4e5cb4e1c86403544173 P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:20.822525 9541 log.cc:1196] T ae76780e32bb4e5cb4e1c86403544173 P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ae76780e32bb4e5cb4e1c86403544173
I20240503 13:59:20.822768 9541 ts_tablet_manager.cc:1909] T ae76780e32bb4e5cb4e1c86403544173 P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:20.823102 9408 raft_consensus.cc:504] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:20.823441 9408 leader_election.cc:288] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:20.823977 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet ae76780e32bb4e5cb4e1c86403544173 (table rebalance_test_table_4 [id=1b29a48257a84332b723b8a0a9ce625b]) successfully deleted
I20240503 13:59:20.824009 9542 ts_tablet_manager.cc:1888] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:20.824070 9542 log.cc:1196] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ae76780e32bb4e5cb4e1c86403544173
I20240503 13:59:20.824151 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e572f8c125044029809ac8c69fa3bad8" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:20.824256 8948 raft_consensus.cc:3037] T e572f8c125044029809ac8c69fa3bad8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:20.824322 9542 ts_tablet_manager.cc:1909] T ae76780e32bb4e5cb4e1c86403544173 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:20.825132 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet ae76780e32bb4e5cb4e1c86403544173 (table rebalance_test_table_4 [id=1b29a48257a84332b723b8a0a9ce625b]) successfully deleted
I20240503 13:59:20.825210 8948 raft_consensus.cc:2454] T e572f8c125044029809ac8c69fa3bad8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:20.825510 9278 leader_election.cc:302] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 0d77752181e944698e809e325e5937e6; no voters:
I20240503 13:59:20.825665 9408 raft_consensus.cc:2781] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:20.825757 9408 raft_consensus.cc:686] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:20.825781 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet 63ec50dbf2624c24bc0e9325ab28bb86 (table rebalance_test_table_4 [id=1b29a48257a84332b723b8a0a9ce625b]) successfully deleted
I20240503 13:59:20.825909 9408 consensus_queue.cc:237] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:20.826534 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e572f8c125044029809ac8c69fa3bad8" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:20.826613 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e572f8c125044029809ac8c69fa3bad8" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:20.826661 9080 raft_consensus.cc:2454] T e572f8c125044029809ac8c69fa3bad8 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:20.827587 6380 catalog_manager.cc:5484] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:20.951952 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:20.955652 9081 raft_consensus.cc:3037] T e572f8c125044029809ac8c69fa3bad8 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:20.955670 8948 raft_consensus.cc:1261] T e572f8c125044029809ac8c69fa3bad8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:20.956096 9657 consensus_queue.cc:1035] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:20.956717 9081 raft_consensus.cc:1261] T e572f8c125044029809ac8c69fa3bad8 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:20.957178 9657 consensus_queue.cc:1035] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:20.958418 6383 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet e572f8c125044029809ac8c69fa3bad8 (table rebalancer_dynamic_table [id=61aae4c8eda1440ba9af9c3cd9eec9a7]). Expected version 1 got 0
I20240503 13:59:20.959565 6383 catalog_manager.cc:5576] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481) does not have the latest schema for tablet e572f8c125044029809ac8c69fa3bad8 (table rebalancer_dynamic_table [id=61aae4c8eda1440ba9af9c3cd9eec9a7]). Expected version 1 got 0
I20240503 13:59:20.959558 9480 tablet.cc:1722] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:20.959878 9480 tablet.cc:1620] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:20.960832 9482 tablet.cc:1722] T e572f8c125044029809ac8c69fa3bad8 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:20.961135 9482 tablet.cc:1620] T e572f8c125044029809ac8c69fa3bad8 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:20.961874 6383 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet e572f8c125044029809ac8c69fa3bad8 (table rebalancer_dynamic_table [id=61aae4c8eda1440ba9af9c3cd9eec9a7]). Expected version 1 got 0
I20240503 13:59:20.962095 9626 tablet.cc:1722] T e572f8c125044029809ac8c69fa3bad8 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:20.962381 9626 tablet.cc:1620] T e572f8c125044029809ac8c69fa3bad8 P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:20.965997 6383 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=61aae4c8eda1440ba9af9c3cd9eec9a7] alter complete (version 1)
I20240503 13:59:21.038810 9551 consensus_queue.cc:1035] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:21.042416 9551 consensus_queue.cc:1035] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:21.067762 6383 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:21.072883 9480 tablet.cc:1722] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:21.073220 9480 tablet.cc:1620] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:21.073459 9482 tablet.cc:1722] T e572f8c125044029809ac8c69fa3bad8 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:21.073748 9482 tablet.cc:1620] T e572f8c125044029809ac8c69fa3bad8 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:21.074254 9626 tablet.cc:1722] T e572f8c125044029809ac8c69fa3bad8 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:21.074524 9626 tablet.cc:1620] T e572f8c125044029809ac8c69fa3bad8 P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:21.078523 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=61aae4c8eda1440ba9af9c3cd9eec9a7] alter complete (version 2)
I20240503 13:59:21.102263 9408 consensus_queue.cc:1035] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:21.107025 9408 consensus_queue.cc:1035] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:21.179567 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:21.184727 9480 tablet.cc:1722] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:21.185746 9626 tablet.cc:1722] T e572f8c125044029809ac8c69fa3bad8 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:21.186575 9482 tablet.cc:1722] T e572f8c125044029809ac8c69fa3bad8 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:21.189450 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=61aae4c8eda1440ba9af9c3cd9eec9a7] alter complete (version 3)
I20240503 13:59:21.217965 9414 consensus_queue.cc:1035] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:21.223397 9414 consensus_queue.cc:1035] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:21.241793 6385 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:21.247514 8928 tablet_service.cc:1467] Processing CreateTablet for tablet ab6fe502a80c410a9b6ee6f5d3d3205e (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=7f5a05b637884ebab6fb86d880c343e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:21.247733 9323 tablet_service.cc:1467] Processing CreateTablet for tablet ab6fe502a80c410a9b6ee6f5d3d3205e (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=7f5a05b637884ebab6fb86d880c343e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:21.247862 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab6fe502a80c410a9b6ee6f5d3d3205e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:21.248016 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab6fe502a80c410a9b6ee6f5d3d3205e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:21.248785 9322 tablet_service.cc:1467] Processing CreateTablet for tablet e29a1f8d9c5c40aaa4023dd3d4204fb5 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=7f5a05b637884ebab6fb86d880c343e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:21.249006 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e29a1f8d9c5c40aaa4023dd3d4204fb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:21.249806 9677 tablet_bootstrap.cc:492] T ab6fe502a80c410a9b6ee6f5d3d3205e P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:21.249982 9058 tablet_service.cc:1467] Processing CreateTablet for tablet ab6fe502a80c410a9b6ee6f5d3d3205e (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=7f5a05b637884ebab6fb86d880c343e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:21.250396 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab6fe502a80c410a9b6ee6f5d3d3205e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:21.250627 6507 tablet_service.cc:1467] Processing CreateTablet for tablet e29a1f8d9c5c40aaa4023dd3d4204fb5 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=7f5a05b637884ebab6fb86d880c343e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:21.250728 9677 tablet_bootstrap.cc:654] T ab6fe502a80c410a9b6ee6f5d3d3205e P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:21.250881 6507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e29a1f8d9c5c40aaa4023dd3d4204fb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:21.251533 9678 tablet_bootstrap.cc:492] T ab6fe502a80c410a9b6ee6f5d3d3205e P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:21.251680 8927 tablet_service.cc:1467] Processing CreateTablet for tablet e29a1f8d9c5c40aaa4023dd3d4204fb5 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=7f5a05b637884ebab6fb86d880c343e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:21.251696 9192 tablet_service.cc:1467] Processing CreateTablet for tablet 80f507bf86ad450f8510b0d7e1a2200c (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=7f5a05b637884ebab6fb86d880c343e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:21.251890 8927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e29a1f8d9c5c40aaa4023dd3d4204fb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:21.251925 9192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80f507bf86ad450f8510b0d7e1a2200c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:21.252784 9678 tablet_bootstrap.cc:654] T ab6fe502a80c410a9b6ee6f5d3d3205e P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:21.253589 9680 tablet_bootstrap.cc:492] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:21.254539 9680 tablet_bootstrap.cc:654] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:21.254483 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 80f507bf86ad450f8510b0d7e1a2200c (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=7f5a05b637884ebab6fb86d880c343e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:21.254703 9678 tablet_bootstrap.cc:492] T ab6fe502a80c410a9b6ee6f5d3d3205e P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:21.254712 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80f507bf86ad450f8510b0d7e1a2200c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:21.254773 9678 ts_tablet_manager.cc:1367] T ab6fe502a80c410a9b6ee6f5d3d3205e P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:21.255167 9678 raft_consensus.cc:348] T ab6fe502a80c410a9b6ee6f5d3d3205e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:21.255292 9678 raft_consensus.cc:374] T ab6fe502a80c410a9b6ee6f5d3d3205e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:21.255334 9678 raft_consensus.cc:729] T ab6fe502a80c410a9b6ee6f5d3d3205e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:21.255460 9678 consensus_queue.cc:260] T ab6fe502a80c410a9b6ee6f5d3d3205e P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:21.255609 9680 tablet_bootstrap.cc:492] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:21.255756 9678 ts_tablet_manager.cc:1398] T ab6fe502a80c410a9b6ee6f5d3d3205e P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:21.255913 9678 tablet_bootstrap.cc:492] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:21.256089 9680 ts_tablet_manager.cc:1367] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 13:59:21.256511 9680 raft_consensus.cc:348] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:21.256618 9680 raft_consensus.cc:374] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:21.256654 9680 raft_consensus.cc:729] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:21.256755 9678 tablet_bootstrap.cc:654] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:21.256933 9680 consensus_queue.cc:260] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:21.257251 9680 ts_tablet_manager.cc:1398] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:21.257398 9680 tablet_bootstrap.cc:492] T 80f507bf86ad450f8510b0d7e1a2200c P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:21.257441 6902 tablet_service.cc:1467] Processing CreateTablet for tablet 80f507bf86ad450f8510b0d7e1a2200c (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=7f5a05b637884ebab6fb86d880c343e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:21.257721 6902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80f507bf86ad450f8510b0d7e1a2200c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:21.258225 9680 tablet_bootstrap.cc:654] T 80f507bf86ad450f8510b0d7e1a2200c P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:21.259243 9680 tablet_bootstrap.cc:492] T 80f507bf86ad450f8510b0d7e1a2200c P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:21.259330 9680 ts_tablet_manager.cc:1367] T 80f507bf86ad450f8510b0d7e1a2200c P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:21.259663 9680 raft_consensus.cc:348] T 80f507bf86ad450f8510b0d7e1a2200c P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:21.259748 9680 raft_consensus.cc:374] T 80f507bf86ad450f8510b0d7e1a2200c P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:21.259783 9680 raft_consensus.cc:729] T 80f507bf86ad450f8510b0d7e1a2200c P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:21.259912 9680 consensus_queue.cc:260] T 80f507bf86ad450f8510b0d7e1a2200c P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:21.260308 9680 ts_tablet_manager.cc:1398] T 80f507bf86ad450f8510b0d7e1a2200c P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:21.261289 9678 tablet_bootstrap.cc:492] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:21.261584 9678 ts_tablet_manager.cc:1367] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:59:21.262205 9686 tablet_bootstrap.cc:492] T 80f507bf86ad450f8510b0d7e1a2200c P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:21.263033 9686 tablet_bootstrap.cc:654] T 80f507bf86ad450f8510b0d7e1a2200c P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:21.263592 9687 tablet_bootstrap.cc:492] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:21.263738 9678 raft_consensus.cc:348] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:21.263849 9678 raft_consensus.cc:374] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:21.263881 9678 raft_consensus.cc:729] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:21.264004 9678 consensus_queue.cc:260] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:21.264250 9678 ts_tablet_manager.cc:1398] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240503 13:59:21.264465 9687 tablet_bootstrap.cc:654] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:21.265179 9686 tablet_bootstrap.cc:492] T 80f507bf86ad450f8510b0d7e1a2200c P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:21.265295 9686 ts_tablet_manager.cc:1367] T 80f507bf86ad450f8510b0d7e1a2200c P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:21.265538 9687 tablet_bootstrap.cc:492] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:21.265625 9687 ts_tablet_manager.cc:1367] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:21.265735 9686 raft_consensus.cc:348] T 80f507bf86ad450f8510b0d7e1a2200c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:21.265856 9686 raft_consensus.cc:374] T 80f507bf86ad450f8510b0d7e1a2200c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:21.265899 9686 raft_consensus.cc:729] T 80f507bf86ad450f8510b0d7e1a2200c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:21.266011 9686 consensus_queue.cc:260] T 80f507bf86ad450f8510b0d7e1a2200c P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:21.266080 9687 raft_consensus.cc:348] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:21.266192 9687 raft_consensus.cc:374] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:21.266225 9687 raft_consensus.cc:729] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:21.266289 9686 ts_tablet_manager.cc:1398] T 80f507bf86ad450f8510b0d7e1a2200c P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:21.266335 9687 consensus_queue.cc:260] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:21.266554 9681 tablet_bootstrap.cc:492] T 80f507bf86ad450f8510b0d7e1a2200c P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:21.267001 9687 ts_tablet_manager.cc:1398] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:21.267447 9681 tablet_bootstrap.cc:654] T 80f507bf86ad450f8510b0d7e1a2200c P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:21.268580 9681 tablet_bootstrap.cc:492] T 80f507bf86ad450f8510b0d7e1a2200c P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:21.268685 9681 ts_tablet_manager.cc:1367] T 80f507bf86ad450f8510b0d7e1a2200c P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:21.269135 9681 raft_consensus.cc:348] T 80f507bf86ad450f8510b0d7e1a2200c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:21.269246 9681 raft_consensus.cc:374] T 80f507bf86ad450f8510b0d7e1a2200c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:21.269280 9681 raft_consensus.cc:729] T 80f507bf86ad450f8510b0d7e1a2200c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:21.269395 9681 consensus_queue.cc:260] T 80f507bf86ad450f8510b0d7e1a2200c P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:21.269685 9681 ts_tablet_manager.cc:1398] T 80f507bf86ad450f8510b0d7e1a2200c P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:21.270381 9677 tablet_bootstrap.cc:492] T ab6fe502a80c410a9b6ee6f5d3d3205e P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:21.270495 9677 ts_tablet_manager.cc:1367] T ab6fe502a80c410a9b6ee6f5d3d3205e P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240503 13:59:21.270992 9677 raft_consensus.cc:348] T ab6fe502a80c410a9b6ee6f5d3d3205e P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:21.271106 9677 raft_consensus.cc:374] T ab6fe502a80c410a9b6ee6f5d3d3205e P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:21.271140 9677 raft_consensus.cc:729] T ab6fe502a80c410a9b6ee6f5d3d3205e P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:21.271250 9677 consensus_queue.cc:260] T ab6fe502a80c410a9b6ee6f5d3d3205e P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:21.271535 9677 ts_tablet_manager.cc:1398] T ab6fe502a80c410a9b6ee6f5d3d3205e P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:21.271747 9677 tablet_bootstrap.cc:492] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:21.272691 9677 tablet_bootstrap.cc:654] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:21.273849 9677 tablet_bootstrap.cc:492] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:21.273933 9677 ts_tablet_manager.cc:1367] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:21.274329 9677 raft_consensus.cc:348] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:21.274464 9677 raft_consensus.cc:374] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:21.274525 9677 raft_consensus.cc:729] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:21.274654 9677 consensus_queue.cc:260] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:21.274914 9677 ts_tablet_manager.cc:1398] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:21.283313 9657 raft_consensus.cc:482] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:21.283406 9657 raft_consensus.cc:504] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:21.283821 9657 leader_election.cc:288] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:21.284409 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e29a1f8d9c5c40aaa4023dd3d4204fb5" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:21.284552 8948 raft_consensus.cc:2454] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:21.284839 9278 leader_election.cc:302] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 0d77752181e944698e809e325e5937e6; no voters:
I20240503 13:59:21.284994 9657 raft_consensus.cc:2781] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:21.285049 9657 raft_consensus.cc:482] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:21.285095 9657 raft_consensus.cc:3037] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:21.286002 9657 raft_consensus.cc:504] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:21.286309 9657 leader_election.cc:288] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:21.286680 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e29a1f8d9c5c40aaa4023dd3d4204fb5" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:21.286792 8948 raft_consensus.cc:3037] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:21.287539 8948 raft_consensus.cc:2454] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:21.287832 9278 leader_election.cc:302] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 0d77752181e944698e809e325e5937e6; no voters:
I20240503 13:59:21.288120 9657 raft_consensus.cc:2781] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:21.288208 9657 raft_consensus.cc:686] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:21.288352 9657 consensus_queue.cc:237] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:21.289216 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e29a1f8d9c5c40aaa4023dd3d4204fb5" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:21.289290 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e29a1f8d9c5c40aaa4023dd3d4204fb5" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:21.289350 6528 raft_consensus.cc:2454] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:21.290062 6385 catalog_manager.cc:5484] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:21.290803 6383 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:21.290894 6383 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:21.292450 6383 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet e572f8c125044029809ac8c69fa3bad8
I20240503 13:59:21.292975 9058 tablet_service.cc:1514] Processing DeleteTablet for tablet e572f8c125044029809ac8c69fa3bad8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:21 UTC) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:21.293053 8927 tablet_service.cc:1514] Processing DeleteTablet for tablet e572f8c125044029809ac8c69fa3bad8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:21 UTC) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:21.293069 9322 tablet_service.cc:1514] Processing DeleteTablet for tablet e572f8c125044029809ac8c69fa3bad8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:21 UTC) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:21.293380 9696 tablet_replica.cc:330] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:21.293503 9695 tablet_replica.cc:330] T e572f8c125044029809ac8c69fa3bad8 P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:21.293506 9696 raft_consensus.cc:2229] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:21.293627 9695 raft_consensus.cc:2229] T e572f8c125044029809ac8c69fa3bad8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:21.293691 9696 raft_consensus.cc:2258] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:21.293742 9695 raft_consensus.cc:2258] T e572f8c125044029809ac8c69fa3bad8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:21.293995 9694 tablet_replica.cc:330] T e572f8c125044029809ac8c69fa3bad8 P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:21.294143 9694 raft_consensus.cc:2229] T e572f8c125044029809ac8c69fa3bad8 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:21.294279 9694 raft_consensus.cc:2258] T e572f8c125044029809ac8c69fa3bad8 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:21.294335 9695 ts_tablet_manager.cc:1875] T e572f8c125044029809ac8c69fa3bad8 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:21.294888 9694 ts_tablet_manager.cc:1875] T e572f8c125044029809ac8c69fa3bad8 P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:21.296139 9695 ts_tablet_manager.cc:1888] T e572f8c125044029809ac8c69fa3bad8 P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:21.296207 9695 log.cc:1196] T e572f8c125044029809ac8c69fa3bad8 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/e572f8c125044029809ac8c69fa3bad8
I20240503 13:59:21.296469 9695 ts_tablet_manager.cc:1909] T e572f8c125044029809ac8c69fa3bad8 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting consensus metadata
I20240503 13:59:21.297425 9696 ts_tablet_manager.cc:1875] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:21.298527 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet e572f8c125044029809ac8c69fa3bad8 (table rebalancer_dynamic_table_new_name [id=61aae4c8eda1440ba9af9c3cd9eec9a7]) successfully deleted
I20240503 13:59:21.299311 9694 ts_tablet_manager.cc:1888] T e572f8c125044029809ac8c69fa3bad8 P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:21.299383 9694 log.cc:1196] T e572f8c125044029809ac8c69fa3bad8 P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/e572f8c125044029809ac8c69fa3bad8
I20240503 13:59:21.299916 9694 ts_tablet_manager.cc:1909] T e572f8c125044029809ac8c69fa3bad8 P 8d574fb4a89d4eea947c859470d44dd4: Deleting consensus metadata
I20240503 13:59:21.300180 9696 ts_tablet_manager.cc:1888] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:21.300246 9696 log.cc:1196] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/e572f8c125044029809ac8c69fa3bad8
I20240503 13:59:21.300509 9696 ts_tablet_manager.cc:1909] T e572f8c125044029809ac8c69fa3bad8 P 0d77752181e944698e809e325e5937e6: Deleting consensus metadata
I20240503 13:59:21.300611 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet e572f8c125044029809ac8c69fa3bad8 (table rebalancer_dynamic_table_new_name [id=61aae4c8eda1440ba9af9c3cd9eec9a7]) successfully deleted
I20240503 13:59:21.301132 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet e572f8c125044029809ac8c69fa3bad8 (table rebalancer_dynamic_table_new_name [id=61aae4c8eda1440ba9af9c3cd9eec9a7]) successfully deleted
I20240503 13:59:21.314667 6383 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240503 13:59:21.314841 6383 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240503 13:59:21.317152 6383 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 927176cab28741f882a2021868e3d744
I20240503 13:59:21.317641 6508 tablet_service.cc:1514] Processing DeleteTablet for tablet 927176cab28741f882a2021868e3d744 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:21 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:21.317680 6383 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 76799d8d81324f898705ee572782261c
I20240503 13:59:21.317771 6640 tablet_service.cc:1514] Processing DeleteTablet for tablet 927176cab28741f882a2021868e3d744 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:21 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:21.317883 9541 tablet_replica.cc:330] T 927176cab28741f882a2021868e3d744 P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:21.317957 9540 tablet_replica.cc:330] T 927176cab28741f882a2021868e3d744 P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:21.318039 9541 raft_consensus.cc:2229] T 927176cab28741f882a2021868e3d744 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:21.318079 6383 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 5e7500549f5c49b1b1e8ab1781bfab4b
I20240503 13:59:21.318090 9540 raft_consensus.cc:2229] T 927176cab28741f882a2021868e3d744 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:21.318122 6771 tablet_service.cc:1514] Processing DeleteTablet for tablet 76799d8d81324f898705ee572782261c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:21 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:21.318183 9541 raft_consensus.cc:2258] T 927176cab28741f882a2021868e3d744 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:21.318238 9540 raft_consensus.cc:2258] T 927176cab28741f882a2021868e3d744 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:21.318732 6640 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e7500549f5c49b1b1e8ab1781bfab4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:21 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:21.318773 9540 ts_tablet_manager.cc:1875] T 927176cab28741f882a2021868e3d744 P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:21.318816 9541 ts_tablet_manager.cc:1875] T 927176cab28741f882a2021868e3d744 P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:21.319479 9542 tablet_replica.cc:330] T 76799d8d81324f898705ee572782261c P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:21.319651 9542 raft_consensus.cc:2229] T 76799d8d81324f898705ee572782261c P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:21.319808 9542 raft_consensus.cc:2258] T 76799d8d81324f898705ee572782261c P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:21.320323 9542 ts_tablet_manager.cc:1875] T 76799d8d81324f898705ee572782261c P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:21.320724 9540 ts_tablet_manager.cc:1888] T 927176cab28741f882a2021868e3d744 P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:21.320773 9540 log.cc:1196] T 927176cab28741f882a2021868e3d744 P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/927176cab28741f882a2021868e3d744
I20240503 13:59:21.320958 9540 ts_tablet_manager.cc:1909] T 927176cab28741f882a2021868e3d744 P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:21.321000 9541 ts_tablet_manager.cc:1888] T 927176cab28741f882a2021868e3d744 P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:21.321065 9541 log.cc:1196] T 927176cab28741f882a2021868e3d744 P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/927176cab28741f882a2021868e3d744
I20240503 13:59:21.321331 9541 ts_tablet_manager.cc:1909] T 927176cab28741f882a2021868e3d744 P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:21.321733 9540 tablet_replica.cc:330] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:21.321837 9540 raft_consensus.cc:2229] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:21.321837 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet 927176cab28741f882a2021868e3d744 (table rebalance_test_table_6 [id=db76abc34823418f9b242b9c4575cb62]) successfully deleted
I20240503 13:59:21.321969 9540 raft_consensus.cc:2258] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:21.322355 9540 ts_tablet_manager.cc:1875] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:21.322376 9542 ts_tablet_manager.cc:1888] T 76799d8d81324f898705ee572782261c P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:21.322424 9542 log.cc:1196] T 76799d8d81324f898705ee572782261c P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/76799d8d81324f898705ee572782261c
I20240503 13:59:21.322608 9542 ts_tablet_manager.cc:1909] T 76799d8d81324f898705ee572782261c P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:21.323192 6902 tablet_service.cc:1514] Processing DeleteTablet for tablet 927176cab28741f882a2021868e3d744 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:21 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:21.323221 6507 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e7500549f5c49b1b1e8ab1781bfab4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:21 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:21.323207 6900 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e7500549f5c49b1b1e8ab1781bfab4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:21 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:21.323382 6508 tablet_service.cc:1514] Processing DeleteTablet for tablet 76799d8d81324f898705ee572782261c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:21 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:21.323493 6901 tablet_service.cc:1514] Processing DeleteTablet for tablet 76799d8d81324f898705ee572782261c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:21 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:21.323531 9697 tablet_replica.cc:330] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:21.323648 9697 raft_consensus.cc:2229] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:21.323849 9697 raft_consensus.cc:2258] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:21.323896 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet 76799d8d81324f898705ee572782261c (table rebalance_test_table_6 [id=db76abc34823418f9b242b9c4575cb62]) successfully deleted
I20240503 13:59:21.324182 9541 tablet_replica.cc:330] T 5e7500549f5c49b1b1e8ab1781bfab4b P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:21.324261 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 927176cab28741f882a2021868e3d744 (table rebalance_test_table_6 [id=db76abc34823418f9b242b9c4575cb62]) successfully deleted
I20240503 13:59:21.324321 9697 ts_tablet_manager.cc:1875] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:21.324337 9541 raft_consensus.cc:2229] T 5e7500549f5c49b1b1e8ab1781bfab4b P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:21.324465 9541 raft_consensus.cc:2258] T 5e7500549f5c49b1b1e8ab1781bfab4b P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:21.324879 9541 ts_tablet_manager.cc:1875] T 5e7500549f5c49b1b1e8ab1781bfab4b P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:21.325955 9697 ts_tablet_manager.cc:1888] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:21.326014 9697 log.cc:1196] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/927176cab28741f882a2021868e3d744
I20240503 13:59:21.326225 9697 ts_tablet_manager.cc:1909] T 927176cab28741f882a2021868e3d744 P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:21.326359 9541 ts_tablet_manager.cc:1888] T 5e7500549f5c49b1b1e8ab1781bfab4b P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:21.326442 9541 log.cc:1196] T 5e7500549f5c49b1b1e8ab1781bfab4b P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/5e7500549f5c49b1b1e8ab1781bfab4b
I20240503 13:59:21.326687 9541 ts_tablet_manager.cc:1909] T 5e7500549f5c49b1b1e8ab1781bfab4b P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:21.326927 9697 tablet_replica.cc:330] T 5e7500549f5c49b1b1e8ab1781bfab4b P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:21.327018 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet 927176cab28741f882a2021868e3d744 (table rebalance_test_table_6 [id=db76abc34823418f9b242b9c4575cb62]) successfully deleted
I20240503 13:59:21.327078 9697 raft_consensus.cc:2229] T 5e7500549f5c49b1b1e8ab1781bfab4b P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:21.327244 9697 raft_consensus.cc:2258] T 5e7500549f5c49b1b1e8ab1781bfab4b P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:21.327391 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 5e7500549f5c49b1b1e8ab1781bfab4b (table rebalance_test_table_6 [id=db76abc34823418f9b242b9c4575cb62]) successfully deleted
I20240503 13:59:21.327431 9541 tablet_replica.cc:330] T 76799d8d81324f898705ee572782261c P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:21.327597 9541 raft_consensus.cc:2229] T 76799d8d81324f898705ee572782261c P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:21.327668 9540 ts_tablet_manager.cc:1888] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:21.327757 9540 log.cc:1196] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/5e7500549f5c49b1b1e8ab1781bfab4b
I20240503 13:59:21.327755 9541 raft_consensus.cc:2258] T 76799d8d81324f898705ee572782261c P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:21.327782 9697 ts_tablet_manager.cc:1875] T 5e7500549f5c49b1b1e8ab1781bfab4b P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:21.328024 9540 ts_tablet_manager.cc:1909] T 5e7500549f5c49b1b1e8ab1781bfab4b P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:21.328255 9541 ts_tablet_manager.cc:1875] T 76799d8d81324f898705ee572782261c P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:21.328747 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet 5e7500549f5c49b1b1e8ab1781bfab4b (table rebalance_test_table_6 [id=db76abc34823418f9b242b9c4575cb62]) successfully deleted
I20240503 13:59:21.329419 9697 ts_tablet_manager.cc:1888] T 5e7500549f5c49b1b1e8ab1781bfab4b P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:21.329486 9697 log.cc:1196] T 5e7500549f5c49b1b1e8ab1781bfab4b P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/5e7500549f5c49b1b1e8ab1781bfab4b
I20240503 13:59:21.329718 9697 ts_tablet_manager.cc:1909] T 5e7500549f5c49b1b1e8ab1781bfab4b P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:21.329911 9541 ts_tablet_manager.cc:1888] T 76799d8d81324f898705ee572782261c P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:21.330029 9541 log.cc:1196] T 76799d8d81324f898705ee572782261c P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/76799d8d81324f898705ee572782261c
I20240503 13:59:21.330305 9541 ts_tablet_manager.cc:1909] T 76799d8d81324f898705ee572782261c P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:21.330322 9697 tablet_replica.cc:330] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:21.330413 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet 5e7500549f5c49b1b1e8ab1781bfab4b (table rebalance_test_table_6 [id=db76abc34823418f9b242b9c4575cb62]) successfully deleted
I20240503 13:59:21.330432 9697 raft_consensus.cc:2229] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:21.330585 9697 raft_consensus.cc:2258] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:21.330870 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 76799d8d81324f898705ee572782261c (table rebalance_test_table_6 [id=db76abc34823418f9b242b9c4575cb62]) successfully deleted
I20240503 13:59:21.330987 9697 ts_tablet_manager.cc:1875] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:21.332437 9697 ts_tablet_manager.cc:1888] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:21.332494 9697 log.cc:1196] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/76799d8d81324f898705ee572782261c
I20240503 13:59:21.332702 9697 ts_tablet_manager.cc:1909] T 76799d8d81324f898705ee572782261c P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:21.333259 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet 76799d8d81324f898705ee572782261c (table rebalance_test_table_6 [id=db76abc34823418f9b242b9c4575cb62]) successfully deleted
I20240503 13:59:21.394248 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:21.399159 8927 tablet_service.cc:1467] Processing CreateTablet for tablet 58bcc0a27c5144a5b95259a442b939ae (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ac2695ba470b48339256c514951c21eb]), partition=
I20240503 13:59:21.399336 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 58bcc0a27c5144a5b95259a442b939ae (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ac2695ba470b48339256c514951c21eb]), partition=
I20240503 13:59:21.399387 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 58bcc0a27c5144a5b95259a442b939ae (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ac2695ba470b48339256c514951c21eb]), partition=
I20240503 13:59:21.399497 8927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58bcc0a27c5144a5b95259a442b939ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:21.399552 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58bcc0a27c5144a5b95259a442b939ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:21.399660 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58bcc0a27c5144a5b95259a442b939ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:21.401546 9687 tablet_bootstrap.cc:492] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:21.401664 9632 raft_consensus.cc:482] T 80f507bf86ad450f8510b0d7e1a2200c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:21.401788 9698 tablet_bootstrap.cc:492] T 58bcc0a27c5144a5b95259a442b939ae P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:21.401762 9632 raft_consensus.cc:504] T 80f507bf86ad450f8510b0d7e1a2200c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:21.402256 9632 leader_election.cc:288] T 80f507bf86ad450f8510b0d7e1a2200c P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:21.402436 9687 tablet_bootstrap.cc:654] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:21.402560 9698 tablet_bootstrap.cc:654] T 58bcc0a27c5144a5b95259a442b939ae P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:21.403053 9677 tablet_bootstrap.cc:492] T 58bcc0a27c5144a5b95259a442b939ae P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:21.403208 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80f507bf86ad450f8510b0d7e1a2200c" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:21.403358 6528 raft_consensus.cc:2454] T 80f507bf86ad450f8510b0d7e1a2200c P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:21.403643 6857 leader_election.cc:302] T 80f507bf86ad450f8510b0d7e1a2200c P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:21.403925 9632 raft_consensus.cc:2781] T 80f507bf86ad450f8510b0d7e1a2200c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:21.403976 9632 raft_consensus.cc:482] T 80f507bf86ad450f8510b0d7e1a2200c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:21.404017 9632 raft_consensus.cc:3037] T 80f507bf86ad450f8510b0d7e1a2200c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:21.404316 9687 tablet_bootstrap.cc:492] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:21.404400 9687 ts_tablet_manager.cc:1367] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:21.404788 9687 raft_consensus.cc:348] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:21.404875 9687 raft_consensus.cc:374] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:21.404852 9632 raft_consensus.cc:504] T 80f507bf86ad450f8510b0d7e1a2200c P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:21.404907 9687 raft_consensus.cc:729] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:21.405053 9687 consensus_queue.cc:260] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:21.405162 9632 leader_election.cc:288] T 80f507bf86ad450f8510b0d7e1a2200c P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:21.405337 9687 ts_tablet_manager.cc:1398] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:21.405431 9677 tablet_bootstrap.cc:654] T 58bcc0a27c5144a5b95259a442b939ae P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:21.405773 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80f507bf86ad450f8510b0d7e1a2200c" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:21.405890 6528 raft_consensus.cc:3037] T 80f507bf86ad450f8510b0d7e1a2200c P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:21.406890 6528 raft_consensus.cc:2454] T 80f507bf86ad450f8510b0d7e1a2200c P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:21.407119 9698 tablet_bootstrap.cc:492] T 58bcc0a27c5144a5b95259a442b939ae P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:21.407202 9698 ts_tablet_manager.cc:1367] T 58bcc0a27c5144a5b95259a442b939ae P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:59:21.407537 6857 leader_election.cc:302] T 80f507bf86ad450f8510b0d7e1a2200c P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:21.407568 9677 tablet_bootstrap.cc:492] T 58bcc0a27c5144a5b95259a442b939ae P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:21.407660 9677 ts_tablet_manager.cc:1367] T 58bcc0a27c5144a5b95259a442b939ae P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:21.407627 9698 raft_consensus.cc:348] T 58bcc0a27c5144a5b95259a442b939ae P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:21.407685 9632 raft_consensus.cc:2781] T 80f507bf86ad450f8510b0d7e1a2200c P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:21.407733 9698 raft_consensus.cc:374] T 58bcc0a27c5144a5b95259a442b939ae P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:21.407770 9632 raft_consensus.cc:686] T 80f507bf86ad450f8510b0d7e1a2200c P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:21.407769 9698 raft_consensus.cc:729] T 58bcc0a27c5144a5b95259a442b939ae P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:21.407891 9632 consensus_queue.cc:237] T 80f507bf86ad450f8510b0d7e1a2200c P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:21.407910 9698 consensus_queue.cc:260] T 58bcc0a27c5144a5b95259a442b939ae P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:21.408130 9677 raft_consensus.cc:348] T 58bcc0a27c5144a5b95259a442b939ae P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:21.408200 9698 ts_tablet_manager.cc:1398] T 58bcc0a27c5144a5b95259a442b939ae P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:21.408229 9677 raft_consensus.cc:374] T 58bcc0a27c5144a5b95259a442b939ae P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:21.408262 9677 raft_consensus.cc:729] T 58bcc0a27c5144a5b95259a442b939ae P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:21.408373 9677 consensus_queue.cc:260] T 58bcc0a27c5144a5b95259a442b939ae P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:21.408622 9677 ts_tablet_manager.cc:1398] T 58bcc0a27c5144a5b95259a442b939ae P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:21.410367 6379 catalog_manager.cc:5484] T 80f507bf86ad450f8510b0d7e1a2200c P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:21.411567 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80f507bf86ad450f8510b0d7e1a2200c" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:21.411718 9205 raft_consensus.cc:2454] T 80f507bf86ad450f8510b0d7e1a2200c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:21.411896 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80f507bf86ad450f8510b0d7e1a2200c" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:21.411991 9211 raft_consensus.cc:3037] T 80f507bf86ad450f8510b0d7e1a2200c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:21.412720 9211 raft_consensus.cc:2454] T 80f507bf86ad450f8510b0d7e1a2200c P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:21.558176 9557 raft_consensus.cc:482] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:21.558296 9557 raft_consensus.cc:504] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:21.559370 9557 leader_election.cc:288] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:21.563764 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab6fe502a80c410a9b6ee6f5d3d3205e" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:21.563937 8948 raft_consensus.cc:2454] T ab6fe502a80c410a9b6ee6f5d3d3205e P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:21.564242 9016 leader_election.cc:302] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:21.564517 9557 raft_consensus.cc:2781] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:21.564482 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab6fe502a80c410a9b6ee6f5d3d3205e" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:21.564599 9557 raft_consensus.cc:482] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:21.564644 9557 raft_consensus.cc:3037] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:21.564642 9343 raft_consensus.cc:2454] T ab6fe502a80c410a9b6ee6f5d3d3205e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:21.565461 9557 raft_consensus.cc:504] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:21.565785 9557 leader_election.cc:288] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:21.566108 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab6fe502a80c410a9b6ee6f5d3d3205e" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:21.566143 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab6fe502a80c410a9b6ee6f5d3d3205e" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:21.566221 8948 raft_consensus.cc:3037] T ab6fe502a80c410a9b6ee6f5d3d3205e P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:21.566365 9343 raft_consensus.cc:3037] T ab6fe502a80c410a9b6ee6f5d3d3205e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:21.567188 8948 raft_consensus.cc:2454] T ab6fe502a80c410a9b6ee6f5d3d3205e P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:21.567540 9343 raft_consensus.cc:2454] T ab6fe502a80c410a9b6ee6f5d3d3205e P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:21.567843 9017 leader_election.cc:302] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:21.567991 9557 raft_consensus.cc:2781] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:21.568210 9557 raft_consensus.cc:686] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 13:59:21.568459 9557 consensus_queue.cc:237] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:21.570760 6380 catalog_manager.cc:5484] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:21.669080 9557 raft_consensus.cc:482] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:21.669198 9557 raft_consensus.cc:504] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:21.669632 9557 leader_election.cc:288] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:21.669974 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bcc0a27c5144a5b95259a442b939ae" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:21.670114 8948 raft_consensus.cc:2454] T 58bcc0a27c5144a5b95259a442b939ae P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:21.670428 9016 leader_election.cc:302] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:21.670594 9557 raft_consensus.cc:2781] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:21.670677 9557 raft_consensus.cc:482] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:21.670727 9557 raft_consensus.cc:3037] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:21.671573 9557 raft_consensus.cc:504] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:21.671927 9557 leader_election.cc:288] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:21.672168 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bcc0a27c5144a5b95259a442b939ae" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:21.672328 8948 raft_consensus.cc:3037] T 58bcc0a27c5144a5b95259a442b939ae P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:21.673353 8948 raft_consensus.cc:2454] T 58bcc0a27c5144a5b95259a442b939ae P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:21.673677 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bcc0a27c5144a5b95259a442b939ae" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:21.673816 6659 raft_consensus.cc:2454] T 58bcc0a27c5144a5b95259a442b939ae P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:21.673908 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bcc0a27c5144a5b95259a442b939ae" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:21.674036 6660 raft_consensus.cc:3037] T 58bcc0a27c5144a5b95259a442b939ae P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:21.674808 6660 raft_consensus.cc:2454] T 58bcc0a27c5144a5b95259a442b939ae P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:21.675112 9017 leader_election.cc:302] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:21.675290 9557 raft_consensus.cc:2781] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:21.675387 9557 raft_consensus.cc:686] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 13:59:21.675529 9557 consensus_queue.cc:237] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:21.677184 6380 catalog_manager.cc:5484] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:21.789484 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:21.794113 6660 raft_consensus.cc:1261] T 58bcc0a27c5144a5b95259a442b939ae P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:21.794274 8948 raft_consensus.cc:1261] T 58bcc0a27c5144a5b95259a442b939ae P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:21.794740 9710 consensus_queue.cc:1035] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:21.794895 9557 consensus_queue.cc:1035] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:21.797624 6380 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet 58bcc0a27c5144a5b95259a442b939ae (table rebalancer_dynamic_table [id=ac2695ba470b48339256c514951c21eb]). Expected version 1 got 0
I20240503 13:59:21.798194 9723 tablet.cc:1722] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:21.798643 9723 tablet.cc:1620] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:21.799036 9718 mvcc.cc:205] Tried to move back new op lower bound from 7023594544304361472 to 7023594543824060416. Current Snapshot: MvccSnapshot[applied={T|T < 7023594544304361472}]
I20240503 13:59:21.799458 6380 catalog_manager.cc:5576] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119) does not have the latest schema for tablet 58bcc0a27c5144a5b95259a442b939ae (table rebalancer_dynamic_table [id=ac2695ba470b48339256c514951c21eb]). Expected version 1 got 0
I20240503 13:59:21.799788 6379 catalog_manager.cc:5576] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481) does not have the latest schema for tablet 58bcc0a27c5144a5b95259a442b939ae (table rebalancer_dynamic_table [id=ac2695ba470b48339256c514951c21eb]). Expected version 1 got 0
I20240503 13:59:21.800498 6379 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet 58bcc0a27c5144a5b95259a442b939ae (table rebalancer_dynamic_table [id=ac2695ba470b48339256c514951c21eb]). Expected version 1 got 0
I20240503 13:59:21.801143 9724 tablet.cc:1722] T 58bcc0a27c5144a5b95259a442b939ae P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:21.801441 9724 tablet.cc:1620] T 58bcc0a27c5144a5b95259a442b939ae P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:21.804293 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=ac2695ba470b48339256c514951c21eb] alter complete (version 1)
I20240503 13:59:21.805011 9726 tablet.cc:1722] T 58bcc0a27c5144a5b95259a442b939ae P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:21.805722 9726 tablet.cc:1620] T 58bcc0a27c5144a5b95259a442b939ae P 2145d15ab2d74c0287839266f1411a43: MemRowSet was empty: no flush needed.
I20240503 13:59:21.806479 6379 catalog_manager.cc:5576] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119) does not have the latest schema for tablet 58bcc0a27c5144a5b95259a442b939ae (table rebalancer_dynamic_table [id=ac2695ba470b48339256c514951c21eb]). Expected version 1 got 0
I20240503 13:59:21.811378 6528 raft_consensus.cc:3037] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:21.812647 9715 consensus_queue.cc:1035] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:21.817523 9657 consensus_queue.cc:1035] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:21.824493 6379 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240503 13:59:21.824625 6379 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240503 13:59:21.826676 6379 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 76e5e28dd13f475695def994b23c4695
I20240503 13:59:21.826931 6379 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 3a51d0527ff841b5b456962d591dcfe7
I20240503 13:59:21.827155 6379 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 5dcc6d09a5a8489a87f1b6bdce05a7ed
I20240503 13:59:21.827271 6901 tablet_service.cc:1514] Processing DeleteTablet for tablet 76e5e28dd13f475695def994b23c4695 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:21 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:21.827325 6508 tablet_service.cc:1514] Processing DeleteTablet for tablet 76e5e28dd13f475695def994b23c4695 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:21 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:21.827471 9697 tablet_replica.cc:330] T 76e5e28dd13f475695def994b23c4695 P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:21.827510 6640 tablet_service.cc:1514] Processing DeleteTablet for tablet 76e5e28dd13f475695def994b23c4695 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:21 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:21.827554 6771 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a51d0527ff841b5b456962d591dcfe7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:21 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:21.827595 9697 raft_consensus.cc:2229] T 76e5e28dd13f475695def994b23c4695 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:21.827708 9541 tablet_replica.cc:330] T 76e5e28dd13f475695def994b23c4695 P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:21.827720 9697 raft_consensus.cc:2258] T 76e5e28dd13f475695def994b23c4695 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:21.827746 9540 tablet_replica.cc:330] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:21.827841 9541 raft_consensus.cc:2229] T 76e5e28dd13f475695def994b23c4695 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:21.827873 9540 raft_consensus.cc:2229] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:21.827981 9541 raft_consensus.cc:2258] T 76e5e28dd13f475695def994b23c4695 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:21.828074 9540 raft_consensus.cc:2258] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:21.828292 9697 ts_tablet_manager.cc:1875] T 76e5e28dd13f475695def994b23c4695 P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:21.828330 9731 tablet_replica.cc:330] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:21.828462 9731 raft_consensus.cc:2229] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:21.828472 9541 ts_tablet_manager.cc:1875] T 76e5e28dd13f475695def994b23c4695 P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:21.828563 9540 ts_tablet_manager.cc:1875] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:21.828665 9731 raft_consensus.cc:2258] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:21.829130 9731 ts_tablet_manager.cc:1875] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:21.830471 9697 ts_tablet_manager.cc:1888] T 76e5e28dd13f475695def994b23c4695 P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:21.830541 9697 log.cc:1196] T 76e5e28dd13f475695def994b23c4695 P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/76e5e28dd13f475695def994b23c4695
I20240503 13:59:21.830574 9541 ts_tablet_manager.cc:1888] T 76e5e28dd13f475695def994b23c4695 P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:21.830634 9541 log.cc:1196] T 76e5e28dd13f475695def994b23c4695 P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/76e5e28dd13f475695def994b23c4695
I20240503 13:59:21.830657 9540 ts_tablet_manager.cc:1888] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:21.830721 9540 log.cc:1196] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/76e5e28dd13f475695def994b23c4695
I20240503 13:59:21.830816 9697 ts_tablet_manager.cc:1909] T 76e5e28dd13f475695def994b23c4695 P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:21.830890 9541 ts_tablet_manager.cc:1909] T 76e5e28dd13f475695def994b23c4695 P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:21.830971 9540 ts_tablet_manager.cc:1909] T 76e5e28dd13f475695def994b23c4695 P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:21.831311 6901 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dcc6d09a5a8489a87f1b6bdce05a7ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:21 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:21.831398 6771 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dcc6d09a5a8489a87f1b6bdce05a7ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:21 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:21.831615 9731 ts_tablet_manager.cc:1888] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:21.831674 9731 log.cc:1196] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/3a51d0527ff841b5b456962d591dcfe7
I20240503 13:59:21.831871 9697 tablet_replica.cc:330] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:21.831959 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet 76e5e28dd13f475695def994b23c4695 (table rebalance_test_table_8 [id=2b50aeb14c1c4234955ff17e9bdb164f]) successfully deleted
I20240503 13:59:21.831992 9697 raft_consensus.cc:2229] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:21.832062 6507 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dcc6d09a5a8489a87f1b6bdce05a7ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:21 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:21.832098 9697 raft_consensus.cc:2258] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:21.832180 6508 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a51d0527ff841b5b456962d591dcfe7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:21 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:21.832393 9731 ts_tablet_manager.cc:1909] T 3a51d0527ff841b5b456962d591dcfe7 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:21.832535 9697 ts_tablet_manager.cc:1875] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:21.832744 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet 76e5e28dd13f475695def994b23c4695 (table rebalance_test_table_8 [id=2b50aeb14c1c4234955ff17e9bdb164f]) successfully deleted
I20240503 13:59:21.832918 6640 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a51d0527ff841b5b456962d591dcfe7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:21 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:21.833190 9731 tablet_replica.cc:330] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:21.833374 9731 raft_consensus.cc:2229] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:21.833411 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet 3a51d0527ff841b5b456962d591dcfe7 (table rebalance_test_table_8 [id=2b50aeb14c1c4234955ff17e9bdb164f]) successfully deleted
I20240503 13:59:21.833513 9731 raft_consensus.cc:2258] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:21.833961 9541 tablet_replica.cc:330] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:21.834000 9731 ts_tablet_manager.cc:1875] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:21.834101 9540 tablet_replica.cc:330] T 3a51d0527ff841b5b456962d591dcfe7 P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:21.834074 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 76e5e28dd13f475695def994b23c4695 (table rebalance_test_table_8 [id=2b50aeb14c1c4234955ff17e9bdb164f]) successfully deleted
I20240503 13:59:21.834257 9541 raft_consensus.cc:2229] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:21.834339 9697 ts_tablet_manager.cc:1888] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:21.834424 9697 log.cc:1196] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/5dcc6d09a5a8489a87f1b6bdce05a7ed
I20240503 13:59:21.834455 9541 raft_consensus.cc:2258] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:21.834677 9697 ts_tablet_manager.cc:1909] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:21.834895 9541 ts_tablet_manager.cc:1875] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:21.835405 9540 raft_consensus.cc:2229] T 3a51d0527ff841b5b456962d591dcfe7 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:21.835569 9540 raft_consensus.cc:2258] T 3a51d0527ff841b5b456962d591dcfe7 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:21.835630 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet 5dcc6d09a5a8489a87f1b6bdce05a7ed (table rebalance_test_table_8 [id=2b50aeb14c1c4234955ff17e9bdb164f]) successfully deleted
I20240503 13:59:21.836086 9540 ts_tablet_manager.cc:1875] T 3a51d0527ff841b5b456962d591dcfe7 P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:21.836985 9731 ts_tablet_manager.cc:1888] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:21.837064 9731 log.cc:1196] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/5dcc6d09a5a8489a87f1b6bdce05a7ed
I20240503 13:59:21.837316 9731 ts_tablet_manager.cc:1909] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:21.837992 9540 ts_tablet_manager.cc:1888] T 3a51d0527ff841b5b456962d591dcfe7 P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:21.838055 9540 log.cc:1196] T 3a51d0527ff841b5b456962d591dcfe7 P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/3a51d0527ff841b5b456962d591dcfe7
I20240503 13:59:21.838018 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet 5dcc6d09a5a8489a87f1b6bdce05a7ed (table rebalance_test_table_8 [id=2b50aeb14c1c4234955ff17e9bdb164f]) successfully deleted
I20240503 13:59:21.838284 9540 ts_tablet_manager.cc:1909] T 3a51d0527ff841b5b456962d591dcfe7 P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:21.839047 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet 3a51d0527ff841b5b456962d591dcfe7 (table rebalance_test_table_8 [id=2b50aeb14c1c4234955ff17e9bdb164f]) successfully deleted
I20240503 13:59:21.839879 9541 ts_tablet_manager.cc:1888] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:21.839942 9541 log.cc:1196] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/5dcc6d09a5a8489a87f1b6bdce05a7ed
I20240503 13:59:21.840167 9541 ts_tablet_manager.cc:1909] T 5dcc6d09a5a8489a87f1b6bdce05a7ed P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:21.840654 9541 tablet_replica.cc:330] T 3a51d0527ff841b5b456962d591dcfe7 P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:21.840754 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 5dcc6d09a5a8489a87f1b6bdce05a7ed (table rebalance_test_table_8 [id=2b50aeb14c1c4234955ff17e9bdb164f]) successfully deleted
I20240503 13:59:21.840809 9541 raft_consensus.cc:2229] T 3a51d0527ff841b5b456962d591dcfe7 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:21.840911 9541 raft_consensus.cc:2258] T 3a51d0527ff841b5b456962d591dcfe7 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:21.841296 9541 ts_tablet_manager.cc:1875] T 3a51d0527ff841b5b456962d591dcfe7 P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:21.842561 9541 ts_tablet_manager.cc:1888] T 3a51d0527ff841b5b456962d591dcfe7 P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:21.842623 9541 log.cc:1196] T 3a51d0527ff841b5b456962d591dcfe7 P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/3a51d0527ff841b5b456962d591dcfe7
I20240503 13:59:21.842798 9541 ts_tablet_manager.cc:1909] T 3a51d0527ff841b5b456962d591dcfe7 P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:21.843523 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 3a51d0527ff841b5b456962d591dcfe7 (table rebalance_test_table_8 [id=2b50aeb14c1c4234955ff17e9bdb164f]) successfully deleted
I20240503 13:59:21.904337 8758 consensus_queue.cc:1035] T 80f507bf86ad450f8510b0d7e1a2200c P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:21.906267 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:21.909615 9632 consensus_queue.cc:1035] T 80f507bf86ad450f8510b0d7e1a2200c P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:21.912551 9723 tablet.cc:1722] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:21.912882 9723 tablet.cc:1620] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:21.913066 9724 tablet.cc:1722] T 58bcc0a27c5144a5b95259a442b939ae P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:21.913367 9724 tablet.cc:1620] T 58bcc0a27c5144a5b95259a442b939ae P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:21.916329 9726 tablet.cc:1722] T 58bcc0a27c5144a5b95259a442b939ae P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:21.916606 9726 tablet.cc:1620] T 58bcc0a27c5144a5b95259a442b939ae P 2145d15ab2d74c0287839266f1411a43: MemRowSet was empty: no flush needed.
I20240503 13:59:21.917766 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=ac2695ba470b48339256c514951c21eb] alter complete (version 2)
I20240503 13:59:22.020913 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:22.026173 9723 tablet.cc:1722] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:22.026643 9724 tablet.cc:1722] T 58bcc0a27c5144a5b95259a442b939ae P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:22.028139 9726 tablet.cc:1722] T 58bcc0a27c5144a5b95259a442b939ae P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:22.031358 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=ac2695ba470b48339256c514951c21eb] alter complete (version 3)
I20240503 13:59:22.081957 9557 consensus_queue.cc:1035] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:22.085467 9725 consensus_queue.cc:1035] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:22.122365 6380 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:22.128136 9058 tablet_service.cc:1467] Processing CreateTablet for tablet aaa93a7fd5df4cb7967d6bdb0c8dd6d1 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=48b6af18e5404209abb657e70e9adedb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:22.128469 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaa93a7fd5df4cb7967d6bdb0c8dd6d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:22.128880 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 3286ea636a5a4d23bf92e8452afd608d (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=48b6af18e5404209abb657e70e9adedb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:22.129165 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3286ea636a5a4d23bf92e8452afd608d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:22.130568 9740 tablet_bootstrap.cc:492] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:22.130718 9192 tablet_service.cc:1467] Processing CreateTablet for tablet aaa93a7fd5df4cb7967d6bdb0c8dd6d1 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=48b6af18e5404209abb657e70e9adedb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:22.130988 9192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaa93a7fd5df4cb7967d6bdb0c8dd6d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:22.131049 9741 tablet_bootstrap.cc:492] T 3286ea636a5a4d23bf92e8452afd608d P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:22.131582 9740 tablet_bootstrap.cc:654] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:22.131632 6901 tablet_service.cc:1467] Processing CreateTablet for tablet aaa93a7fd5df4cb7967d6bdb0c8dd6d1 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=48b6af18e5404209abb657e70e9adedb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:22.131920 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaa93a7fd5df4cb7967d6bdb0c8dd6d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:22.131920 9741 tablet_bootstrap.cc:654] T 3286ea636a5a4d23bf92e8452afd608d P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:22.133129 9744 tablet_bootstrap.cc:492] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:22.133273 9741 tablet_bootstrap.cc:492] T 3286ea636a5a4d23bf92e8452afd608d P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:22.133338 9741 ts_tablet_manager.cc:1367] T 3286ea636a5a4d23bf92e8452afd608d P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:22.133633 9741 raft_consensus.cc:348] T 3286ea636a5a4d23bf92e8452afd608d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:22.133724 9741 raft_consensus.cc:374] T 3286ea636a5a4d23bf92e8452afd608d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:22.133756 9741 raft_consensus.cc:729] T 3286ea636a5a4d23bf92e8452afd608d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:22.133867 9741 consensus_queue.cc:260] T 3286ea636a5a4d23bf92e8452afd608d P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:22.134061 9744 tablet_bootstrap.cc:654] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:22.135155 8927 tablet_service.cc:1467] Processing CreateTablet for tablet 3286ea636a5a4d23bf92e8452afd608d (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=48b6af18e5404209abb657e70e9adedb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:22.135367 9744 tablet_bootstrap.cc:492] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:22.135314 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 568c2e7206f9496baa22432b72c021b1 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=48b6af18e5404209abb657e70e9adedb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:22.135440 9744 ts_tablet_manager.cc:1367] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:22.135557 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568c2e7206f9496baa22432b72c021b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:22.135529 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 568c2e7206f9496baa22432b72c021b1 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=48b6af18e5404209abb657e70e9adedb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:22.135758 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568c2e7206f9496baa22432b72c021b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:22.135753 9191 tablet_service.cc:1467] Processing CreateTablet for tablet 568c2e7206f9496baa22432b72c021b1 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=48b6af18e5404209abb657e70e9adedb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:22.135888 9744 raft_consensus.cc:348] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:22.135962 9191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568c2e7206f9496baa22432b72c021b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:22.135982 9744 raft_consensus.cc:374] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:22.136011 9744 raft_consensus.cc:729] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:22.136124 9744 consensus_queue.cc:260] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:22.136394 9744 ts_tablet_manager.cc:1398] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:22.136889 9740 tablet_bootstrap.cc:492] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:22.136979 9740 ts_tablet_manager.cc:1367] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:59:22.137480 9740 raft_consensus.cc:348] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:22.137583 9740 raft_consensus.cc:374] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:22.137612 9740 raft_consensus.cc:729] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:22.137732 9740 consensus_queue.cc:260] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:22.137979 9740 ts_tablet_manager.cc:1398] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:22.138299 8927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3286ea636a5a4d23bf92e8452afd608d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:22.138603 9744 tablet_bootstrap.cc:492] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:22.139855 9744 tablet_bootstrap.cc:654] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:22.140460 6383 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:22.140595 6383 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:22.140838 9741 ts_tablet_manager.cc:1398] T 3286ea636a5a4d23bf92e8452afd608d P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 13:59:22.141077 9740 tablet_bootstrap.cc:492] T 568c2e7206f9496baa22432b72c021b1 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:22.141446 9747 tablet_bootstrap.cc:492] T 568c2e7206f9496baa22432b72c021b1 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:22.142336 9740 tablet_bootstrap.cc:654] T 568c2e7206f9496baa22432b72c021b1 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:22.142380 9747 tablet_bootstrap.cc:654] T 568c2e7206f9496baa22432b72c021b1 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:22.143105 9192 tablet_service.cc:1467] Processing CreateTablet for tablet 3286ea636a5a4d23bf92e8452afd608d (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=48b6af18e5404209abb657e70e9adedb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:22.143369 9192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3286ea636a5a4d23bf92e8452afd608d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:22.143854 9740 tablet_bootstrap.cc:492] T 568c2e7206f9496baa22432b72c021b1 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:22.143927 9740 ts_tablet_manager.cc:1367] T 568c2e7206f9496baa22432b72c021b1 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:22.144351 9740 raft_consensus.cc:348] T 568c2e7206f9496baa22432b72c021b1 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:22.144450 9740 raft_consensus.cc:374] T 568c2e7206f9496baa22432b72c021b1 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:22.144491 9740 raft_consensus.cc:729] T 568c2e7206f9496baa22432b72c021b1 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:22.144651 9740 consensus_queue.cc:260] T 568c2e7206f9496baa22432b72c021b1 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:22.144922 9740 ts_tablet_manager.cc:1398] T 568c2e7206f9496baa22432b72c021b1 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:22.145444 9557 raft_consensus.cc:482] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:22.145526 9557 raft_consensus.cc:504] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:22.145668 6383 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 58bcc0a27c5144a5b95259a442b939ae
I20240503 13:59:22.145951 9557 leader_election.cc:288] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:22.146222 8927 tablet_service.cc:1514] Processing DeleteTablet for tablet 58bcc0a27c5144a5b95259a442b939ae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:22 UTC) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:22.146312 6640 tablet_service.cc:1514] Processing DeleteTablet for tablet 58bcc0a27c5144a5b95259a442b939ae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:22 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:22.146481 9540 tablet_replica.cc:330] T 58bcc0a27c5144a5b95259a442b939ae P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:22.146616 9540 raft_consensus.cc:2229] T 58bcc0a27c5144a5b95259a442b939ae P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:22.146756 9540 raft_consensus.cc:2258] T 58bcc0a27c5144a5b95259a442b939ae P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:22.146867 9744 tablet_bootstrap.cc:492] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:22.146953 9744 ts_tablet_manager.cc:1367] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:59:22.147238 9058 tablet_service.cc:1514] Processing DeleteTablet for tablet 58bcc0a27c5144a5b95259a442b939ae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:22 UTC) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:22.147500 9744 raft_consensus.cc:348] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:22.147620 9744 raft_consensus.cc:374] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:22.147686 9744 raft_consensus.cc:729] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:22.147833 9744 consensus_queue.cc:260] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:22.147908 9540 ts_tablet_manager.cc:1875] T 58bcc0a27c5144a5b95259a442b939ae P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:22.149150 9751 tablet_replica.cc:330] T 58bcc0a27c5144a5b95259a442b939ae P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:22.149358 9751 raft_consensus.cc:2229] T 58bcc0a27c5144a5b95259a442b939ae P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:22.149632 9751 raft_consensus.cc:2258] T 58bcc0a27c5144a5b95259a442b939ae P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:22.150105 9540 ts_tablet_manager.cc:1888] T 58bcc0a27c5144a5b95259a442b939ae P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:22.150173 9540 log.cc:1196] T 58bcc0a27c5144a5b95259a442b939ae P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/58bcc0a27c5144a5b95259a442b939ae
I20240503 13:59:22.150354 9751 ts_tablet_manager.cc:1875] T 58bcc0a27c5144a5b95259a442b939ae P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:22.150424 9540 ts_tablet_manager.cc:1909] T 58bcc0a27c5144a5b95259a442b939ae P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:22.150427 9744 ts_tablet_manager.cc:1398] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:22.150585 9744 tablet_bootstrap.cc:492] T 3286ea636a5a4d23bf92e8452afd608d P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:22.152837 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaa93a7fd5df4cb7967d6bdb0c8dd6d1" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:22.152961 9211 raft_consensus.cc:2454] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:22.153277 9016 leader_election.cc:302] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:22.153370 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet 58bcc0a27c5144a5b95259a442b939ae (table rebalancer_dynamic_table_new_name [id=ac2695ba470b48339256c514951c21eb]) successfully deleted
I20240503 13:59:22.153455 9557 raft_consensus.cc:2781] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:22.153499 9557 raft_consensus.cc:482] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:22.153541 9557 raft_consensus.cc:3037] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:22.154084 9744 tablet_bootstrap.cc:654] T 3286ea636a5a4d23bf92e8452afd608d P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:22.154474 9557 raft_consensus.cc:504] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:22.154827 9557 leader_election.cc:288] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:22.155056 9712 raft_consensus.cc:482] T 3286ea636a5a4d23bf92e8452afd608d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:22.155057 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaa93a7fd5df4cb7967d6bdb0c8dd6d1" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:22.155148 9211 raft_consensus.cc:3037] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:22.155134 9712 raft_consensus.cc:504] T 3286ea636a5a4d23bf92e8452afd608d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:22.155731 9712 leader_election.cc:288] T 3286ea636a5a4d23bf92e8452afd608d P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:22.156054 9211 raft_consensus.cc:2454] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:22.156450 9016 leader_election.cc:302] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:22.156550 9751 ts_tablet_manager.cc:1888] T 58bcc0a27c5144a5b95259a442b939ae P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:22.156630 9751 log.cc:1196] T 58bcc0a27c5144a5b95259a442b939ae P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/58bcc0a27c5144a5b95259a442b939ae
I20240503 13:59:22.156679 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3286ea636a5a4d23bf92e8452afd608d" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
W20240503 13:59:22.157130 6595 leader_election.cc:341] T 3286ea636a5a4d23bf92e8452afd608d P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:59:22.157160 9557 raft_consensus.cc:2781] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:22.157243 9557 raft_consensus.cc:686] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 13:59:22.157352 9751 ts_tablet_manager.cc:1909] T 58bcc0a27c5144a5b95259a442b939ae P 06ca8dbc004c42b99faa527a4e6f9545: Deleting consensus metadata
I20240503 13:59:22.157392 9557 consensus_queue.cc:237] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:22.158049 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 58bcc0a27c5144a5b95259a442b939ae (table rebalancer_dynamic_table_new_name [id=ac2695ba470b48339256c514951c21eb]) successfully deleted
I20240503 13:59:22.158259 9745 tablet_bootstrap.cc:492] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:22.158492 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaa93a7fd5df4cb7967d6bdb0c8dd6d1" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:22.158552 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaa93a7fd5df4cb7967d6bdb0c8dd6d1" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:22.159148 6380 catalog_manager.cc:5484] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:22.159833 9016 leader_election.cc:341] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): Illegal state: must be running to vote when last-logged opid is not known
W20240503 13:59:22.160068 9016 leader_election.cc:341] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:59:22.160825 9745 tablet_bootstrap.cc:654] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:22.161825 9744 tablet_bootstrap.cc:492] T 3286ea636a5a4d23bf92e8452afd608d P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:22.161912 9744 ts_tablet_manager.cc:1367] T 3286ea636a5a4d23bf92e8452afd608d P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 13:59:22.162351 9744 raft_consensus.cc:348] T 3286ea636a5a4d23bf92e8452afd608d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:22.162510 9744 raft_consensus.cc:374] T 3286ea636a5a4d23bf92e8452afd608d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:22.162572 9744 raft_consensus.cc:729] T 3286ea636a5a4d23bf92e8452afd608d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:22.162701 9744 consensus_queue.cc:260] T 3286ea636a5a4d23bf92e8452afd608d P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:22.162972 9744 ts_tablet_manager.cc:1398] T 3286ea636a5a4d23bf92e8452afd608d P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:22.163570 9745 tablet_bootstrap.cc:492] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:22.163619 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3286ea636a5a4d23bf92e8452afd608d" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:22.163657 9745 ts_tablet_manager.cc:1367] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
W20240503 13:59:22.164094 6595 leader_election.cc:341] T 3286ea636a5a4d23bf92e8452afd608d P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:59:22.164099 9745 raft_consensus.cc:348] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:22.164181 6595 leader_election.cc:302] T 3286ea636a5a4d23bf92e8452afd608d P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43; no voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735
I20240503 13:59:22.164206 9745 raft_consensus.cc:374] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:22.164242 9745 raft_consensus.cc:729] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:22.164335 9712 raft_consensus.cc:2726] T 3286ea636a5a4d23bf92e8452afd608d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 13:59:22.164357 9745 consensus_queue.cc:260] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:22.165489 9745 ts_tablet_manager.cc:1398] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:22.165964 9755 tablet_replica.cc:330] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:22.166123 9755 raft_consensus.cc:2229] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:22.166198 9747 tablet_bootstrap.cc:492] T 568c2e7206f9496baa22432b72c021b1 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:22.166288 9747 ts_tablet_manager.cc:1367] T 568c2e7206f9496baa22432b72c021b1 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240503 13:59:22.166421 9755 raft_consensus.cc:2258] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:22.166754 9747 raft_consensus.cc:348] T 568c2e7206f9496baa22432b72c021b1 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:22.166839 9747 raft_consensus.cc:374] T 568c2e7206f9496baa22432b72c021b1 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:22.166865 9747 raft_consensus.cc:729] T 568c2e7206f9496baa22432b72c021b1 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:22.166973 9747 consensus_queue.cc:260] T 568c2e7206f9496baa22432b72c021b1 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:22.167210 9747 ts_tablet_manager.cc:1398] T 568c2e7206f9496baa22432b72c021b1 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:22.167414 9747 tablet_bootstrap.cc:492] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:22.167543 9755 ts_tablet_manager.cc:1875] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:22.168365 9747 tablet_bootstrap.cc:654] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:22.169435 9747 tablet_bootstrap.cc:492] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:22.169518 9747 ts_tablet_manager.cc:1367] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:22.169917 9747 raft_consensus.cc:348] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:22.169998 9747 raft_consensus.cc:374] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:22.170030 9747 raft_consensus.cc:729] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:22.170147 9747 consensus_queue.cc:260] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:22.170421 9747 ts_tablet_manager.cc:1398] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:22.170868 9755 ts_tablet_manager.cc:1888] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:22.170933 9755 log.cc:1196] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/58bcc0a27c5144a5b95259a442b939ae
I20240503 13:59:22.171273 9755 ts_tablet_manager.cc:1909] T 58bcc0a27c5144a5b95259a442b939ae P 8d574fb4a89d4eea947c859470d44dd4: Deleting consensus metadata
I20240503 13:59:22.171896 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet 58bcc0a27c5144a5b95259a442b939ae (table rebalancer_dynamic_table_new_name [id=ac2695ba470b48339256c514951c21eb]) successfully deleted
I20240503 13:59:22.197434 9610 raft_consensus.cc:482] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:22.197567 9610 raft_consensus.cc:504] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:22.197940 9610 leader_election.cc:288] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:22.198361 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568c2e7206f9496baa22432b72c021b1" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:22.198396 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568c2e7206f9496baa22432b72c021b1" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:22.198498 8948 raft_consensus.cc:2454] T 568c2e7206f9496baa22432b72c021b1 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:22.198549 9081 raft_consensus.cc:2454] T 568c2e7206f9496baa22432b72c021b1 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:22.198814 9147 leader_election.cc:302] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:22.199003 9610 raft_consensus.cc:2781] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:22.199059 9610 raft_consensus.cc:482] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:22.199108 9610 raft_consensus.cc:3037] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:22.200158 9610 raft_consensus.cc:504] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:22.200495 9610 leader_election.cc:288] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Requested vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:22.200974 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568c2e7206f9496baa22432b72c021b1" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:22.201114 9081 raft_consensus.cc:3037] T 568c2e7206f9496baa22432b72c021b1 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:22.201293 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568c2e7206f9496baa22432b72c021b1" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:22.201408 8948 raft_consensus.cc:3037] T 568c2e7206f9496baa22432b72c021b1 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:22.201925 9081 raft_consensus.cc:2454] T 568c2e7206f9496baa22432b72c021b1 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:22.202090 8948 raft_consensus.cc:2454] T 568c2e7206f9496baa22432b72c021b1 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:22.202297 9147 leader_election.cc:302] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:22.202529 9610 raft_consensus.cc:2781] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:22.202733 9610 raft_consensus.cc:686] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 13:59:22.202901 9610 consensus_queue.cc:237] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:22.204466 6383 catalog_manager.cc:5484] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:22.207602 9717 raft_consensus.cc:482] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:22.207695 9717 raft_consensus.cc:504] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:22.208098 9717 leader_election.cc:288] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:22.211988 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3286ea636a5a4d23bf92e8452afd608d" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:22.212150 9211 raft_consensus.cc:2454] T 3286ea636a5a4d23bf92e8452afd608d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:22.212436 8883 leader_election.cc:302] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:22.212487 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3286ea636a5a4d23bf92e8452afd608d" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:22.212615 9717 raft_consensus.cc:2781] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:22.212632 6660 raft_consensus.cc:2454] T 3286ea636a5a4d23bf92e8452afd608d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:22.212675 9717 raft_consensus.cc:482] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:22.212711 9717 raft_consensus.cc:3037] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:22.213521 9717 raft_consensus.cc:504] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:22.213858 9717 leader_election.cc:288] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:22.214181 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3286ea636a5a4d23bf92e8452afd608d" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:22.214190 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3286ea636a5a4d23bf92e8452afd608d" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:22.214300 6660 raft_consensus.cc:3037] T 3286ea636a5a4d23bf92e8452afd608d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:22.214323 9211 raft_consensus.cc:3037] T 3286ea636a5a4d23bf92e8452afd608d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:22.215179 6660 raft_consensus.cc:2454] T 3286ea636a5a4d23bf92e8452afd608d P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:22.215180 9211 raft_consensus.cc:2454] T 3286ea636a5a4d23bf92e8452afd608d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:22.215469 8884 leader_election.cc:302] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 2145d15ab2d74c0287839266f1411a43; no voters:
I20240503 13:59:22.215644 9717 raft_consensus.cc:2781] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:22.215713 9717 raft_consensus.cc:686] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 13:59:22.215855 9717 consensus_queue.cc:237] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:22.217449 6383 catalog_manager.cc:5484] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:22.247130 9733 raft_consensus.cc:482] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:22.247097 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:22.247243 9733 raft_consensus.cc:504] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:22.247613 9733 leader_election.cc:288] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:22.248019 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaa93a7fd5df4cb7967d6bdb0c8dd6d1" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:22.248085 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaa93a7fd5df4cb7967d6bdb0c8dd6d1" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:22.248255 9211 raft_consensus.cc:2379] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f58065db70b64ef8b1dc499aa6ae4460 in current term 1: Already voted for candidate 8d574fb4a89d4eea947c859470d44dd4 in this term.
I20240503 13:59:22.248526 6857 leader_election.cc:302] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 8d574fb4a89d4eea947c859470d44dd4, ab1d21935d6049888eb0ceccc3c9d735
I20240503 13:59:22.248769 9733 raft_consensus.cc:3037] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:22.249738 9733 raft_consensus.cc:2726] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 13:59:22.252283 9192 tablet_service.cc:1467] Processing CreateTablet for tablet 9c0c876590a84600be15cbe38038bd98 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=44366f4c770943d790ce493cb1484076]), partition=
I20240503 13:59:22.252408 9322 tablet_service.cc:1467] Processing CreateTablet for tablet 9c0c876590a84600be15cbe38038bd98 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=44366f4c770943d790ce493cb1484076]), partition=
I20240503 13:59:22.252533 9192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c0c876590a84600be15cbe38038bd98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:22.252660 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c0c876590a84600be15cbe38038bd98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:22.254351 9744 tablet_bootstrap.cc:492] T 9c0c876590a84600be15cbe38038bd98 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:22.254479 9768 tablet_bootstrap.cc:492] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:22.254601 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 9c0c876590a84600be15cbe38038bd98 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=44366f4c770943d790ce493cb1484076]), partition=
I20240503 13:59:22.254886 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c0c876590a84600be15cbe38038bd98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:22.255427 9744 tablet_bootstrap.cc:654] T 9c0c876590a84600be15cbe38038bd98 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:22.255461 9768 tablet_bootstrap.cc:654] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:22.256989 9768 tablet_bootstrap.cc:492] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:22.257083 9768 ts_tablet_manager.cc:1367] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:59:22.257370 9771 tablet_bootstrap.cc:492] T 9c0c876590a84600be15cbe38038bd98 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:22.257735 9768 raft_consensus.cc:348] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:22.258258 9771 tablet_bootstrap.cc:654] T 9c0c876590a84600be15cbe38038bd98 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:22.258299 9744 tablet_bootstrap.cc:492] T 9c0c876590a84600be15cbe38038bd98 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:22.258281 9768 raft_consensus.cc:374] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:22.258335 9768 raft_consensus.cc:729] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:22.258379 9744 ts_tablet_manager.cc:1367] T 9c0c876590a84600be15cbe38038bd98 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 13:59:22.258769 9768 consensus_queue.cc:260] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:22.258951 9744 raft_consensus.cc:348] T 9c0c876590a84600be15cbe38038bd98 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:22.259047 9744 raft_consensus.cc:374] T 9c0c876590a84600be15cbe38038bd98 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:22.259079 9744 raft_consensus.cc:729] T 9c0c876590a84600be15cbe38038bd98 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:22.259199 9744 consensus_queue.cc:260] T 9c0c876590a84600be15cbe38038bd98 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:22.259338 9771 tablet_bootstrap.cc:492] T 9c0c876590a84600be15cbe38038bd98 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:22.259346 9768 ts_tablet_manager.cc:1398] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:22.259418 9771 ts_tablet_manager.cc:1367] T 9c0c876590a84600be15cbe38038bd98 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:22.259459 9744 ts_tablet_manager.cc:1398] T 9c0c876590a84600be15cbe38038bd98 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:22.259789 9771 raft_consensus.cc:348] T 9c0c876590a84600be15cbe38038bd98 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:22.259863 9771 raft_consensus.cc:374] T 9c0c876590a84600be15cbe38038bd98 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:22.259882 9771 raft_consensus.cc:729] T 9c0c876590a84600be15cbe38038bd98 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:22.259977 9771 consensus_queue.cc:260] T 9c0c876590a84600be15cbe38038bd98 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:22.260228 9771 ts_tablet_manager.cc:1398] T 9c0c876590a84600be15cbe38038bd98 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:22.328842 6383 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240503 13:59:22.328992 6383 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240503 13:59:22.331113 6383 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 3699694bf8dc46439ccf9b0aa1468369
I20240503 13:59:22.331924 6383 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet d014ee0844784387ade205aca2b676ca
I20240503 13:59:22.332073 6771 tablet_service.cc:1514] Processing DeleteTablet for tablet 3699694bf8dc46439ccf9b0aa1468369 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:22 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:22.332072 6640 tablet_service.cc:1514] Processing DeleteTablet for tablet 3699694bf8dc46439ccf9b0aa1468369 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:22 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:22.332259 9540 tablet_replica.cc:330] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:22.332264 6508 tablet_service.cc:1514] Processing DeleteTablet for tablet 3699694bf8dc46439ccf9b0aa1468369 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:22 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:22.332391 6383 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet ade8477f7bd04858a350c7849e1c5af9
I20240503 13:59:22.332391 9540 raft_consensus.cc:2229] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:22.332412 6901 tablet_service.cc:1514] Processing DeleteTablet for tablet d014ee0844784387ade205aca2b676ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:22 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:22.332455 9731 tablet_replica.cc:330] T 3699694bf8dc46439ccf9b0aa1468369 P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:22.332573 9697 tablet_replica.cc:330] T d014ee0844784387ade205aca2b676ca P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:22.332572 9731 raft_consensus.cc:2229] T 3699694bf8dc46439ccf9b0aa1468369 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:22.332619 9540 raft_consensus.cc:2258] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:22.332715 9731 raft_consensus.cc:2258] T 3699694bf8dc46439ccf9b0aa1468369 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:22.332715 9697 raft_consensus.cc:2229] T d014ee0844784387ade205aca2b676ca P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:22.332861 9697 raft_consensus.cc:2258] T d014ee0844784387ade205aca2b676ca P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:22.333110 9540 ts_tablet_manager.cc:1875] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:22.333205 9731 ts_tablet_manager.cc:1875] T 3699694bf8dc46439ccf9b0aa1468369 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:22.333283 9697 ts_tablet_manager.cc:1875] T d014ee0844784387ade205aca2b676ca P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:22.335338 9541 tablet_replica.cc:330] T 3699694bf8dc46439ccf9b0aa1468369 P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:22.335507 9541 raft_consensus.cc:2229] T 3699694bf8dc46439ccf9b0aa1468369 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:22.335511 9697 ts_tablet_manager.cc:1888] T d014ee0844784387ade205aca2b676ca P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:22.335608 9697 log.cc:1196] T d014ee0844784387ade205aca2b676ca P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/d014ee0844784387ade205aca2b676ca
I20240503 13:59:22.335625 6640 tablet_service.cc:1514] Processing DeleteTablet for tablet ade8477f7bd04858a350c7849e1c5af9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:22 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:22.335670 9541 raft_consensus.cc:2258] T 3699694bf8dc46439ccf9b0aa1468369 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:22.335930 9697 ts_tablet_manager.cc:1909] T d014ee0844784387ade205aca2b676ca P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:22.336030 9540 ts_tablet_manager.cc:1888] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:22.336091 9540 log.cc:1196] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/3699694bf8dc46439ccf9b0aa1468369
I20240503 13:59:22.336256 9541 ts_tablet_manager.cc:1875] T 3699694bf8dc46439ccf9b0aa1468369 P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:22.336349 9540 ts_tablet_manager.cc:1909] T 3699694bf8dc46439ccf9b0aa1468369 P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:22.336403 6769 tablet_service.cc:1514] Processing DeleteTablet for tablet ade8477f7bd04858a350c7849e1c5af9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:22 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:22.336944 9540 tablet_replica.cc:330] T ade8477f7bd04858a350c7849e1c5af9 P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:22.337040 6507 tablet_service.cc:1514] Processing DeleteTablet for tablet ade8477f7bd04858a350c7849e1c5af9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:22 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:22.337297 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet 3699694bf8dc46439ccf9b0aa1468369 (table rebalance_test_table_10 [id=bc0e9003ae9c41d9947f04fddb7235b3]) successfully deleted
I20240503 13:59:22.337566 9731 ts_tablet_manager.cc:1888] T 3699694bf8dc46439ccf9b0aa1468369 P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:22.337639 9731 log.cc:1196] T 3699694bf8dc46439ccf9b0aa1468369 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/3699694bf8dc46439ccf9b0aa1468369
I20240503 13:59:22.337714 9540 raft_consensus.cc:2229] T ade8477f7bd04858a350c7849e1c5af9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:22.337846 9540 raft_consensus.cc:2258] T ade8477f7bd04858a350c7849e1c5af9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:22.337903 9731 ts_tablet_manager.cc:1909] T 3699694bf8dc46439ccf9b0aa1468369 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:22.338092 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet d014ee0844784387ade205aca2b676ca (table rebalance_test_table_10 [id=bc0e9003ae9c41d9947f04fddb7235b3]) successfully deleted
I20240503 13:59:22.338308 9540 ts_tablet_manager.cc:1875] T ade8477f7bd04858a350c7849e1c5af9 P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:22.338424 9541 ts_tablet_manager.cc:1888] T 3699694bf8dc46439ccf9b0aa1468369 P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:22.338487 9541 log.cc:1196] T 3699694bf8dc46439ccf9b0aa1468369 P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/3699694bf8dc46439ccf9b0aa1468369
I20240503 13:59:22.338570 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet 3699694bf8dc46439ccf9b0aa1468369 (table rebalance_test_table_10 [id=bc0e9003ae9c41d9947f04fddb7235b3]) successfully deleted
I20240503 13:59:22.338490 9731 tablet_replica.cc:330] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:22.338913 9731 raft_consensus.cc:2229] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:22.339027 6771 tablet_service.cc:1514] Processing DeleteTablet for tablet d014ee0844784387ade205aca2b676ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:22 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:22.339151 9731 raft_consensus.cc:2258] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:22.339157 6508 tablet_service.cc:1514] Processing DeleteTablet for tablet d014ee0844784387ade205aca2b676ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:22 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:22.339377 9541 ts_tablet_manager.cc:1909] T 3699694bf8dc46439ccf9b0aa1468369 P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:22.339648 9731 ts_tablet_manager.cc:1875] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:22.339991 9541 tablet_replica.cc:330] T ade8477f7bd04858a350c7849e1c5af9 P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:22.340121 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 3699694bf8dc46439ccf9b0aa1468369 (table rebalance_test_table_10 [id=bc0e9003ae9c41d9947f04fddb7235b3]) successfully deleted
I20240503 13:59:22.340284 9540 ts_tablet_manager.cc:1888] T ade8477f7bd04858a350c7849e1c5af9 P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:22.340355 9540 log.cc:1196] T ade8477f7bd04858a350c7849e1c5af9 P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/ade8477f7bd04858a350c7849e1c5af9
I20240503 13:59:22.340364 9541 raft_consensus.cc:2229] T ade8477f7bd04858a350c7849e1c5af9 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:22.340508 9541 raft_consensus.cc:2258] T ade8477f7bd04858a350c7849e1c5af9 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:22.340663 9540 ts_tablet_manager.cc:1909] T ade8477f7bd04858a350c7849e1c5af9 P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:22.341020 9541 ts_tablet_manager.cc:1875] T ade8477f7bd04858a350c7849e1c5af9 P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:22.341292 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet ade8477f7bd04858a350c7849e1c5af9 (table rebalance_test_table_10 [id=bc0e9003ae9c41d9947f04fddb7235b3]) successfully deleted
I20240503 13:59:22.341521 9731 ts_tablet_manager.cc:1888] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:22.341580 9731 log.cc:1196] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ade8477f7bd04858a350c7849e1c5af9
I20240503 13:59:22.341800 9731 ts_tablet_manager.cc:1909] T ade8477f7bd04858a350c7849e1c5af9 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:22.342625 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet ade8477f7bd04858a350c7849e1c5af9 (table rebalance_test_table_10 [id=bc0e9003ae9c41d9947f04fddb7235b3]) successfully deleted
I20240503 13:59:22.342558 9731 tablet_replica.cc:330] T d014ee0844784387ade205aca2b676ca P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:22.342854 9731 raft_consensus.cc:2229] T d014ee0844784387ade205aca2b676ca P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:22.342937 9541 ts_tablet_manager.cc:1888] T ade8477f7bd04858a350c7849e1c5af9 P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:22.342967 9731 raft_consensus.cc:2258] T d014ee0844784387ade205aca2b676ca P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:22.343008 9541 log.cc:1196] T ade8477f7bd04858a350c7849e1c5af9 P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ade8477f7bd04858a350c7849e1c5af9
I20240503 13:59:22.343221 9541 ts_tablet_manager.cc:1909] T ade8477f7bd04858a350c7849e1c5af9 P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:22.343374 9731 ts_tablet_manager.cc:1875] T d014ee0844784387ade205aca2b676ca P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:22.343760 9657 raft_consensus.cc:482] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:22.343817 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet ade8477f7bd04858a350c7849e1c5af9 (table rebalance_test_table_10 [id=bc0e9003ae9c41d9947f04fddb7235b3]) successfully deleted
I20240503 13:59:22.343744 9541 tablet_replica.cc:330] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:22.343870 9657 raft_consensus.cc:504] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:22.344029 9541 raft_consensus.cc:2229] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:22.344220 9541 raft_consensus.cc:2258] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:22.344273 9657 leader_election.cc:288] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:22.344708 9541 ts_tablet_manager.cc:1875] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:22.344738 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c0c876590a84600be15cbe38038bd98" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:22.344731 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c0c876590a84600be15cbe38038bd98" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:22.344878 6527 raft_consensus.cc:2454] T 9c0c876590a84600be15cbe38038bd98 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:22.344882 9205 raft_consensus.cc:2454] T 9c0c876590a84600be15cbe38038bd98 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:22.345188 9278 leader_election.cc:302] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:22.345482 9731 ts_tablet_manager.cc:1888] T d014ee0844784387ade205aca2b676ca P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:22.345474 9657 raft_consensus.cc:2781] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:22.345528 9731 log.cc:1196] T d014ee0844784387ade205aca2b676ca P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/d014ee0844784387ade205aca2b676ca
I20240503 13:59:22.345531 9657 raft_consensus.cc:482] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:22.345582 9657 raft_consensus.cc:3037] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:22.345703 9731 ts_tablet_manager.cc:1909] T d014ee0844784387ade205aca2b676ca P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:22.346503 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet d014ee0844784387ade205aca2b676ca (table rebalance_test_table_10 [id=bc0e9003ae9c41d9947f04fddb7235b3]) successfully deleted
I20240503 13:59:22.346525 9541 ts_tablet_manager.cc:1888] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:22.346488 9657 raft_consensus.cc:504] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:22.346583 9541 log.cc:1196] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/d014ee0844784387ade205aca2b676ca
I20240503 13:59:22.346863 9541 ts_tablet_manager.cc:1909] T d014ee0844784387ade205aca2b676ca P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:22.346874 9657 leader_election.cc:288] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:22.347188 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c0c876590a84600be15cbe38038bd98" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:22.347206 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c0c876590a84600be15cbe38038bd98" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:22.347307 6527 raft_consensus.cc:3037] T 9c0c876590a84600be15cbe38038bd98 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:22.347326 9205 raft_consensus.cc:3037] T 9c0c876590a84600be15cbe38038bd98 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:22.347983 6527 raft_consensus.cc:2454] T 9c0c876590a84600be15cbe38038bd98 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:22.348150 9205 raft_consensus.cc:2454] T 9c0c876590a84600be15cbe38038bd98 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:22.348244 9278 leader_election.cc:302] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:22.348394 9657 raft_consensus.cc:2781] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:22.348556 9657 raft_consensus.cc:686] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:22.348541 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet d014ee0844784387ade205aca2b676ca (table rebalance_test_table_10 [id=bc0e9003ae9c41d9947f04fddb7235b3]) successfully deleted
I20240503 13:59:22.348702 9657 consensus_queue.cc:237] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:22.350236 6383 catalog_manager.cc:5484] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:22.475891 6383 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:22.479677 6527 raft_consensus.cc:1261] T 9c0c876590a84600be15cbe38038bd98 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:22.479739 9205 raft_consensus.cc:1261] T 9c0c876590a84600be15cbe38038bd98 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:22.480113 9775 consensus_queue.cc:1035] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:22.480230 9773 consensus_queue.cc:1035] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:22.482555 6383 catalog_manager.cc:5576] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091) does not have the latest schema for tablet 9c0c876590a84600be15cbe38038bd98 (table rebalancer_dynamic_table [id=44366f4c770943d790ce493cb1484076]). Expected version 1 got 0
I20240503 13:59:22.483206 6380 catalog_manager.cc:5576] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021) does not have the latest schema for tablet 9c0c876590a84600be15cbe38038bd98 (table rebalancer_dynamic_table [id=44366f4c770943d790ce493cb1484076]). Expected version 1 got 0
I20240503 13:59:22.483927 9781 tablet.cc:1722] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:22.484746 9781 tablet.cc:1620] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:22.485102 6380 catalog_manager.cc:5576] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091) does not have the latest schema for tablet 9c0c876590a84600be15cbe38038bd98 (table rebalancer_dynamic_table [id=44366f4c770943d790ce493cb1484076]). Expected version 1 got 0
I20240503 13:59:22.486300 6383 catalog_manager.cc:5576] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931) does not have the latest schema for tablet 9c0c876590a84600be15cbe38038bd98 (table rebalancer_dynamic_table [id=44366f4c770943d790ce493cb1484076]). Expected version 1 got 0
I20240503 13:59:22.486821 9782 tablet.cc:1722] T 9c0c876590a84600be15cbe38038bd98 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:22.487159 6380 catalog_manager.cc:5576] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021) does not have the latest schema for tablet 9c0c876590a84600be15cbe38038bd98 (table rebalancer_dynamic_table [id=44366f4c770943d790ce493cb1484076]). Expected version 1 got 0
I20240503 13:59:22.487457 9783 tablet.cc:1722] T 9c0c876590a84600be15cbe38038bd98 P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:22.487192 9782 tablet.cc:1620] T 9c0c876590a84600be15cbe38038bd98 P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:22.489470 6385 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=44366f4c770943d790ce493cb1484076] alter complete (version 1)
I20240503 13:59:22.489852 9783 tablet.cc:1620] T 9c0c876590a84600be15cbe38038bd98 P 5f46e19644fd4f4b825c86ba6af768da: MemRowSet was empty: no flush needed.
I20240503 13:59:22.535780 9725 consensus_queue.cc:1035] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:22.539211 9710 consensus_queue.cc:1035] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:22.591854 6385 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:22.597182 9781 tablet.cc:1722] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:22.597904 9781 tablet.cc:1620] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:22.598395 9783 tablet.cc:1722] T 9c0c876590a84600be15cbe38038bd98 P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:22.598824 9783 tablet.cc:1620] T 9c0c876590a84600be15cbe38038bd98 P 5f46e19644fd4f4b825c86ba6af768da: MemRowSet was empty: no flush needed.
I20240503 13:59:22.598274 9782 tablet.cc:1722] T 9c0c876590a84600be15cbe38038bd98 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:22.599537 9782 tablet.cc:1620] T 9c0c876590a84600be15cbe38038bd98 P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:22.603811 6385 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=44366f4c770943d790ce493cb1484076] alter complete (version 2)
I20240503 13:59:22.646837 9610 consensus_queue.cc:1035] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:22.649775 9791 consensus_queue.cc:1035] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:22.707875 6385 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:22.713155 9781 tablet.cc:1722] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:22.713521 9783 tablet.cc:1722] T 9c0c876590a84600be15cbe38038bd98 P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:22.713644 9782 tablet.cc:1722] T 9c0c876590a84600be15cbe38038bd98 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:22.717974 6385 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=44366f4c770943d790ce493cb1484076] alter complete (version 3)
I20240503 13:59:22.724889 9717 consensus_queue.cc:1035] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:22.727903 9796 consensus_queue.cc:1035] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:22.731860 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:22.740571 8927 tablet_service.cc:1467] Processing CreateTablet for tablet e92eaea6e9c8493690b749f162949ac4 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=9e8aad8638e34703aac55865282ca799]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:22.740870 8927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e92eaea6e9c8493690b749f162949ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:22.741037 6771 tablet_service.cc:1467] Processing CreateTablet for tablet 8d25b0d68b7643cbbbdeb787cac7a1a4 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=9e8aad8638e34703aac55865282ca799]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:22.741202 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 08c307c79c4b4901be7a4e878c70c4bf (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=9e8aad8638e34703aac55865282ca799]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:22.741307 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d25b0d68b7643cbbbdeb787cac7a1a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:22.741436 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08c307c79c4b4901be7a4e878c70c4bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:22.743530 9192 tablet_service.cc:1467] Processing CreateTablet for tablet 8d25b0d68b7643cbbbdeb787cac7a1a4 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=9e8aad8638e34703aac55865282ca799]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:22.743539 9191 tablet_service.cc:1467] Processing CreateTablet for tablet e92eaea6e9c8493690b749f162949ac4 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=9e8aad8638e34703aac55865282ca799]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:22.743826 9192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d25b0d68b7643cbbbdeb787cac7a1a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:22.743945 9191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e92eaea6e9c8493690b749f162949ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:22.744014 8927 tablet_service.cc:1467] Processing CreateTablet for tablet 08c307c79c4b4901be7a4e878c70c4bf (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=9e8aad8638e34703aac55865282ca799]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:22.744246 8927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08c307c79c4b4901be7a4e878c70c4bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:22.744447 9801 tablet_bootstrap.cc:492] T 08c307c79c4b4901be7a4e878c70c4bf P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:22.746112 9190 tablet_service.cc:1467] Processing CreateTablet for tablet 08c307c79c4b4901be7a4e878c70c4bf (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=9e8aad8638e34703aac55865282ca799]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:22.746213 9801 tablet_bootstrap.cc:654] T 08c307c79c4b4901be7a4e878c70c4bf P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:22.746336 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08c307c79c4b4901be7a4e878c70c4bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:22.746398 9322 tablet_service.cc:1467] Processing CreateTablet for tablet 8d25b0d68b7643cbbbdeb787cac7a1a4 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=9e8aad8638e34703aac55865282ca799]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:22.746577 9799 tablet_bootstrap.cc:492] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:22.746654 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d25b0d68b7643cbbbdeb787cac7a1a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:22.746852 9800 tablet_bootstrap.cc:492] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:22.747496 9799 tablet_bootstrap.cc:654] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:22.747706 9800 tablet_bootstrap.cc:654] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:22.748387 9744 tablet_bootstrap.cc:492] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:22.748487 9799 tablet_bootstrap.cc:492] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:22.748553 9799 ts_tablet_manager.cc:1367] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:22.748867 9323 tablet_service.cc:1467] Processing CreateTablet for tablet e92eaea6e9c8493690b749f162949ac4 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=9e8aad8638e34703aac55865282ca799]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:22.748973 9799 raft_consensus.cc:348] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:22.749073 9799 raft_consensus.cc:374] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:22.749099 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e92eaea6e9c8493690b749f162949ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:22.749105 9799 raft_consensus.cc:729] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:22.749253 9744 tablet_bootstrap.cc:654] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:22.749279 9799 consensus_queue.cc:260] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:22.749480 9800 tablet_bootstrap.cc:492] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:22.749572 9800 ts_tablet_manager.cc:1367] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:59:22.749576 9799 ts_tablet_manager.cc:1398] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:22.749743 9799 tablet_bootstrap.cc:492] T 08c307c79c4b4901be7a4e878c70c4bf P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:22.750008 9800 raft_consensus.cc:348] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:22.750118 9800 raft_consensus.cc:374] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:22.750154 9800 raft_consensus.cc:729] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:22.750275 9800 consensus_queue.cc:260] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:22.750541 9800 ts_tablet_manager.cc:1398] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:22.750559 9799 tablet_bootstrap.cc:654] T 08c307c79c4b4901be7a4e878c70c4bf P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:22.751047 9768 tablet_bootstrap.cc:492] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:22.751936 9768 tablet_bootstrap.cc:654] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:22.753222 9799 tablet_bootstrap.cc:492] T 08c307c79c4b4901be7a4e878c70c4bf P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:22.753288 9799 ts_tablet_manager.cc:1367] T 08c307c79c4b4901be7a4e878c70c4bf P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 13:59:22.753441 9768 tablet_bootstrap.cc:492] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:22.753525 9768 ts_tablet_manager.cc:1367] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:22.753778 9799 raft_consensus.cc:348] T 08c307c79c4b4901be7a4e878c70c4bf P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:22.753856 9799 raft_consensus.cc:374] T 08c307c79c4b4901be7a4e878c70c4bf P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:22.753881 9799 raft_consensus.cc:729] T 08c307c79c4b4901be7a4e878c70c4bf P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:22.754001 9799 consensus_queue.cc:260] T 08c307c79c4b4901be7a4e878c70c4bf P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:22.754014 9768 raft_consensus.cc:348] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:22.754113 9768 raft_consensus.cc:374] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:22.754148 9768 raft_consensus.cc:729] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:22.754287 9768 consensus_queue.cc:260] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:22.754560 9768 ts_tablet_manager.cc:1398] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:22.754731 9768 tablet_bootstrap.cc:492] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:22.754758 9799 ts_tablet_manager.cc:1398] T 08c307c79c4b4901be7a4e878c70c4bf P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:22.755585 9768 tablet_bootstrap.cc:654] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:22.756611 9768 tablet_bootstrap.cc:492] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:22.756644 9801 tablet_bootstrap.cc:492] T 08c307c79c4b4901be7a4e878c70c4bf P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:22.756713 9768 ts_tablet_manager.cc:1367] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:22.756726 9801 ts_tablet_manager.cc:1367] T 08c307c79c4b4901be7a4e878c70c4bf P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 13:59:22.756829 9744 tablet_bootstrap.cc:492] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:22.756897 9744 ts_tablet_manager.cc:1367] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 13:59:22.757094 9768 raft_consensus.cc:348] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:22.757185 9768 raft_consensus.cc:374] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:22.757153 9801 raft_consensus.cc:348] T 08c307c79c4b4901be7a4e878c70c4bf P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:22.757215 9768 raft_consensus.cc:729] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:22.757243 9801 raft_consensus.cc:374] T 08c307c79c4b4901be7a4e878c70c4bf P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:22.757275 9801 raft_consensus.cc:729] T 08c307c79c4b4901be7a4e878c70c4bf P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:22.757282 9744 raft_consensus.cc:348] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:22.757365 9744 raft_consensus.cc:374] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:22.757390 9744 raft_consensus.cc:729] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:22.757365 9768 consensus_queue.cc:260] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:22.757386 9801 consensus_queue.cc:260] T 08c307c79c4b4901be7a4e878c70c4bf P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:22.757506 9744 consensus_queue.cc:260] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:22.757658 9768 ts_tablet_manager.cc:1398] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:22.757676 9801 ts_tablet_manager.cc:1398] T 08c307c79c4b4901be7a4e878c70c4bf P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:22.757723 9744 ts_tablet_manager.cc:1398] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:22.757867 9744 tablet_bootstrap.cc:492] T e92eaea6e9c8493690b749f162949ac4 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:22.758704 9744 tablet_bootstrap.cc:654] T e92eaea6e9c8493690b749f162949ac4 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:22.760864 9744 tablet_bootstrap.cc:492] T e92eaea6e9c8493690b749f162949ac4 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:22.760933 9744 ts_tablet_manager.cc:1367] T e92eaea6e9c8493690b749f162949ac4 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:22.762053 9744 raft_consensus.cc:348] T e92eaea6e9c8493690b749f162949ac4 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:22.762181 9744 raft_consensus.cc:374] T e92eaea6e9c8493690b749f162949ac4 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:22.762221 9744 raft_consensus.cc:729] T e92eaea6e9c8493690b749f162949ac4 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:22.762414 9744 consensus_queue.cc:260] T e92eaea6e9c8493690b749f162949ac4 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:22.763128 9744 ts_tablet_manager.cc:1398] T e92eaea6e9c8493690b749f162949ac4 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:22.763319 9744 tablet_bootstrap.cc:492] T 08c307c79c4b4901be7a4e878c70c4bf P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:22.764236 9744 tablet_bootstrap.cc:654] T 08c307c79c4b4901be7a4e878c70c4bf P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:22.765345 9744 tablet_bootstrap.cc:492] T 08c307c79c4b4901be7a4e878c70c4bf P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:22.765426 9744 ts_tablet_manager.cc:1367] T 08c307c79c4b4901be7a4e878c70c4bf P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:22.765777 9744 raft_consensus.cc:348] T 08c307c79c4b4901be7a4e878c70c4bf P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:22.765911 9744 raft_consensus.cc:374] T 08c307c79c4b4901be7a4e878c70c4bf P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:22.765997 9744 raft_consensus.cc:729] T 08c307c79c4b4901be7a4e878c70c4bf P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:22.766142 9744 consensus_queue.cc:260] T 08c307c79c4b4901be7a4e878c70c4bf P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:22.766906 9744 ts_tablet_manager.cc:1398] T 08c307c79c4b4901be7a4e878c70c4bf P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:22.821339 6383 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:22.821493 6383 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:22.823719 6383 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 9c0c876590a84600be15cbe38038bd98
I20240503 13:59:22.824290 9323 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c0c876590a84600be15cbe38038bd98 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:22 UTC) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:22.824317 6508 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c0c876590a84600be15cbe38038bd98 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:22 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:22.824615 9541 tablet_replica.cc:330] T 9c0c876590a84600be15cbe38038bd98 P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:22.824792 9541 raft_consensus.cc:2229] T 9c0c876590a84600be15cbe38038bd98 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:22.824777 9190 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c0c876590a84600be15cbe38038bd98 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:22 UTC) from {username='slave'} at 127.0.0.1:57960
I20240503 13:59:22.824918 9541 raft_consensus.cc:2258] T 9c0c876590a84600be15cbe38038bd98 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:22.825237 9811 tablet_replica.cc:330] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:22.825310 9812 tablet_replica.cc:330] T 9c0c876590a84600be15cbe38038bd98 P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 13:59:22.825425 9811 raft_consensus.cc:2229] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:22.825462 9812 raft_consensus.cc:2229] T 9c0c876590a84600be15cbe38038bd98 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:22.825569 9541 ts_tablet_manager.cc:1875] T 9c0c876590a84600be15cbe38038bd98 P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:22.825616 9812 raft_consensus.cc:2258] T 9c0c876590a84600be15cbe38038bd98 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:22.825700 9811 raft_consensus.cc:2258] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:22.826195 9811 ts_tablet_manager.cc:1875] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:22.826195 9812 ts_tablet_manager.cc:1875] T 9c0c876590a84600be15cbe38038bd98 P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:22.827705 9541 ts_tablet_manager.cc:1888] T 9c0c876590a84600be15cbe38038bd98 P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:22.827775 9541 log.cc:1196] T 9c0c876590a84600be15cbe38038bd98 P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/9c0c876590a84600be15cbe38038bd98
I20240503 13:59:22.827998 9541 ts_tablet_manager.cc:1909] T 9c0c876590a84600be15cbe38038bd98 P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:22.828297 9812 ts_tablet_manager.cc:1888] T 9c0c876590a84600be15cbe38038bd98 P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:22.828325 9811 ts_tablet_manager.cc:1888] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:22.828377 9812 log.cc:1196] T 9c0c876590a84600be15cbe38038bd98 P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/9c0c876590a84600be15cbe38038bd98
I20240503 13:59:22.828384 9811 log.cc:1196] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/9c0c876590a84600be15cbe38038bd98
I20240503 13:59:22.828672 9812 ts_tablet_manager.cc:1909] T 9c0c876590a84600be15cbe38038bd98 P ab1d21935d6049888eb0ceccc3c9d735: Deleting consensus metadata
I20240503 13:59:22.828855 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 9c0c876590a84600be15cbe38038bd98 (table rebalancer_dynamic_table_new_name [id=44366f4c770943d790ce493cb1484076]) successfully deleted
I20240503 13:59:22.828982 9811 ts_tablet_manager.cc:1909] T 9c0c876590a84600be15cbe38038bd98 P 0d77752181e944698e809e325e5937e6: Deleting consensus metadata
I20240503 13:59:22.829453 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet 9c0c876590a84600be15cbe38038bd98 (table rebalancer_dynamic_table_new_name [id=44366f4c770943d790ce493cb1484076]) successfully deleted
I20240503 13:59:22.829608 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet 9c0c876590a84600be15cbe38038bd98 (table rebalancer_dynamic_table_new_name [id=44366f4c770943d790ce493cb1484076]) successfully deleted
I20240503 13:59:22.837690 6383 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240503 13:59:22.837842 6383 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240503 13:59:22.840354 6383 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 629248af74c2468ba52fd6104e2d98d9
I20240503 13:59:22.840598 6383 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 9890415b2d5a40a3b2f4434779ebf7eb
I20240503 13:59:22.840921 6901 tablet_service.cc:1514] Processing DeleteTablet for tablet 629248af74c2468ba52fd6104e2d98d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:22 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:22.841037 6383 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 94960e06ff44463285360fa7f10f1b68
I20240503 13:59:22.841027 6771 tablet_service.cc:1514] Processing DeleteTablet for tablet 9890415b2d5a40a3b2f4434779ebf7eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:22 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:22.841174 6640 tablet_service.cc:1514] Processing DeleteTablet for tablet 629248af74c2468ba52fd6104e2d98d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:22 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:22.841245 9731 tablet_replica.cc:330] T 9890415b2d5a40a3b2f4434779ebf7eb P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:22.841266 6639 tablet_service.cc:1514] Processing DeleteTablet for tablet 9890415b2d5a40a3b2f4434779ebf7eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:22 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:22.841354 9813 tablet_replica.cc:330] T 629248af74c2468ba52fd6104e2d98d9 P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:22.841363 9540 tablet_replica.cc:330] T 629248af74c2468ba52fd6104e2d98d9 P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:22.841364 9731 raft_consensus.cc:2229] T 9890415b2d5a40a3b2f4434779ebf7eb P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:22.841473 9731 raft_consensus.cc:2258] T 9890415b2d5a40a3b2f4434779ebf7eb P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:22.841475 9813 raft_consensus.cc:2229] T 629248af74c2468ba52fd6104e2d98d9 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:22.841488 9540 raft_consensus.cc:2229] T 629248af74c2468ba52fd6104e2d98d9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:22.841605 9540 raft_consensus.cc:2258] T 629248af74c2468ba52fd6104e2d98d9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:22.841593 9813 raft_consensus.cc:2258] T 629248af74c2468ba52fd6104e2d98d9 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:22.841961 9731 ts_tablet_manager.cc:1875] T 9890415b2d5a40a3b2f4434779ebf7eb P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:22.842020 9540 ts_tablet_manager.cc:1875] T 629248af74c2468ba52fd6104e2d98d9 P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:22.842128 9813 ts_tablet_manager.cc:1875] T 629248af74c2468ba52fd6104e2d98d9 P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:22.843703 9791 raft_consensus.cc:482] T 08c307c79c4b4901be7a4e878c70c4bf P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:22.843737 9540 ts_tablet_manager.cc:1888] T 629248af74c2468ba52fd6104e2d98d9 P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:22.843796 9540 log.cc:1196] T 629248af74c2468ba52fd6104e2d98d9 P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/629248af74c2468ba52fd6104e2d98d9
I20240503 13:59:22.843804 9791 raft_consensus.cc:504] T 08c307c79c4b4901be7a4e878c70c4bf P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:22.844053 9540 ts_tablet_manager.cc:1909] T 629248af74c2468ba52fd6104e2d98d9 P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:22.844118 9731 ts_tablet_manager.cc:1888] T 9890415b2d5a40a3b2f4434779ebf7eb P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:22.844188 9731 log.cc:1196] T 9890415b2d5a40a3b2f4434779ebf7eb P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/9890415b2d5a40a3b2f4434779ebf7eb
I20240503 13:59:22.844209 9813 ts_tablet_manager.cc:1888] T 629248af74c2468ba52fd6104e2d98d9 P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:22.844269 9813 log.cc:1196] T 629248af74c2468ba52fd6104e2d98d9 P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/629248af74c2468ba52fd6104e2d98d9
I20240503 13:59:22.844277 9791 leader_election.cc:288] T 08c307c79c4b4901be7a4e878c70c4bf P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:22.844512 9813 ts_tablet_manager.cc:1909] T 629248af74c2468ba52fd6104e2d98d9 P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:22.844624 6771 tablet_service.cc:1514] Processing DeleteTablet for tablet 94960e06ff44463285360fa7f10f1b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:22 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:22.844733 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c307c79c4b4901be7a4e878c70c4bf" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:22.844851 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c307c79c4b4901be7a4e878c70c4bf" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:22.844900 8948 raft_consensus.cc:2454] T 08c307c79c4b4901be7a4e878c70c4bf P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:22.844995 9081 raft_consensus.cc:2454] T 08c307c79c4b4901be7a4e878c70c4bf P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:22.845191 9147 leader_election.cc:302] T 08c307c79c4b4901be7a4e878c70c4bf P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:22.845204 6901 tablet_service.cc:1514] Processing DeleteTablet for tablet 94960e06ff44463285360fa7f10f1b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:22 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:22.845357 9791 raft_consensus.cc:2781] T 08c307c79c4b4901be7a4e878c70c4bf P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:22.845407 9791 raft_consensus.cc:482] T 08c307c79c4b4901be7a4e878c70c4bf P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:22.845443 9791 raft_consensus.cc:3037] T 08c307c79c4b4901be7a4e878c70c4bf P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:22.845453 6508 tablet_service.cc:1514] Processing DeleteTablet for tablet 629248af74c2468ba52fd6104e2d98d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:22 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:22.845453 6507 tablet_service.cc:1514] Processing DeleteTablet for tablet 9890415b2d5a40a3b2f4434779ebf7eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:22 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:22.845475 9731 ts_tablet_manager.cc:1909] T 9890415b2d5a40a3b2f4434779ebf7eb P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:22.845682 9541 tablet_replica.cc:330] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:22.845813 9541 raft_consensus.cc:2229] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:22.846014 9541 raft_consensus.cc:2258] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:22.846226 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet 9890415b2d5a40a3b2f4434779ebf7eb (table rebalance_test_table_12 [id=e0c6bd9b27794a2887eccea514c01058]) successfully deleted
I20240503 13:59:22.846314 9813 tablet_replica.cc:330] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:22.846369 9791 raft_consensus.cc:504] T 08c307c79c4b4901be7a4e878c70c4bf P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:22.846431 9813 raft_consensus.cc:2229] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:22.846513 9541 ts_tablet_manager.cc:1875] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:22.846693 6507 tablet_service.cc:1514] Processing DeleteTablet for tablet 94960e06ff44463285360fa7f10f1b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:22 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:22.846695 9791 leader_election.cc:288] T 08c307c79c4b4901be7a4e878c70c4bf P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Requested vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:22.846798 9813 raft_consensus.cc:2258] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:22.846603 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet 629248af74c2468ba52fd6104e2d98d9 (table rebalance_test_table_12 [id=e0c6bd9b27794a2887eccea514c01058]) successfully deleted
I20240503 13:59:22.847134 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c307c79c4b4901be7a4e878c70c4bf" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:22.847234 9731 tablet_replica.cc:330] T 94960e06ff44463285360fa7f10f1b68 P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:22.847239 9081 raft_consensus.cc:3037] T 08c307c79c4b4901be7a4e878c70c4bf P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:22.847329 9813 ts_tablet_manager.cc:1875] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:22.847368 9731 raft_consensus.cc:2229] T 94960e06ff44463285360fa7f10f1b68 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:22.847474 9731 raft_consensus.cc:2258] T 94960e06ff44463285360fa7f10f1b68 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:22.847918 9731 ts_tablet_manager.cc:1875] T 94960e06ff44463285360fa7f10f1b68 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:22.848393 9081 raft_consensus.cc:2454] T 08c307c79c4b4901be7a4e878c70c4bf P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:22.848646 9541 ts_tablet_manager.cc:1888] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:22.848709 9541 log.cc:1196] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/629248af74c2468ba52fd6104e2d98d9
I20240503 13:59:22.848691 9147 leader_election.cc:302] T 08c307c79c4b4901be7a4e878c70c4bf P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:22.848963 9541 ts_tablet_manager.cc:1909] T 629248af74c2468ba52fd6104e2d98d9 P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:22.848944 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c307c79c4b4901be7a4e878c70c4bf" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:22.849051 8948 raft_consensus.cc:3037] T 08c307c79c4b4901be7a4e878c70c4bf P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:22.849464 9813 ts_tablet_manager.cc:1888] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:22.849526 9813 log.cc:1196] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/94960e06ff44463285360fa7f10f1b68
I20240503 13:59:22.849653 9541 tablet_replica.cc:330] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:22.849776 9541 raft_consensus.cc:2229] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:22.849797 9813 ts_tablet_manager.cc:1909] T 94960e06ff44463285360fa7f10f1b68 P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:22.849915 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 629248af74c2468ba52fd6104e2d98d9 (table rebalance_test_table_12 [id=e0c6bd9b27794a2887eccea514c01058]) successfully deleted
I20240503 13:59:22.849949 9541 raft_consensus.cc:2258] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:22.850044 9731 ts_tablet_manager.cc:1888] T 94960e06ff44463285360fa7f10f1b68 P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:22.850072 8948 raft_consensus.cc:2454] T 08c307c79c4b4901be7a4e878c70c4bf P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:22.850106 9731 log.cc:1196] T 94960e06ff44463285360fa7f10f1b68 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/94960e06ff44463285360fa7f10f1b68
I20240503 13:59:22.850417 9541 ts_tablet_manager.cc:1875] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:22.850451 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet 94960e06ff44463285360fa7f10f1b68 (table rebalance_test_table_12 [id=e0c6bd9b27794a2887eccea514c01058]) successfully deleted
I20240503 13:59:22.850484 9731 ts_tablet_manager.cc:1909] T 94960e06ff44463285360fa7f10f1b68 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:22.850970 9791 raft_consensus.cc:2781] T 08c307c79c4b4901be7a4e878c70c4bf P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:22.851024 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet 94960e06ff44463285360fa7f10f1b68 (table rebalance_test_table_12 [id=e0c6bd9b27794a2887eccea514c01058]) successfully deleted
I20240503 13:59:22.851066 9791 raft_consensus.cc:686] T 08c307c79c4b4901be7a4e878c70c4bf P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 13:59:22.851181 9791 consensus_queue.cc:237] T 08c307c79c4b4901be7a4e878c70c4bf P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:22.852381 9540 tablet_replica.cc:330] T 9890415b2d5a40a3b2f4434779ebf7eb P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:22.852407 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet 629248af74c2468ba52fd6104e2d98d9 (table rebalance_test_table_12 [id=e0c6bd9b27794a2887eccea514c01058]) successfully deleted
I20240503 13:59:22.852805 9540 raft_consensus.cc:2229] T 9890415b2d5a40a3b2f4434779ebf7eb P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:22.852960 9540 raft_consensus.cc:2258] T 9890415b2d5a40a3b2f4434779ebf7eb P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:22.852910 6383 catalog_manager.cc:5484] T 08c307c79c4b4901be7a4e878c70c4bf P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:22.853437 9540 ts_tablet_manager.cc:1875] T 9890415b2d5a40a3b2f4434779ebf7eb P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:22.854071 9541 ts_tablet_manager.cc:1888] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:22.854147 9541 log.cc:1196] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/9890415b2d5a40a3b2f4434779ebf7eb
I20240503 13:59:22.854442 9541 ts_tablet_manager.cc:1909] T 9890415b2d5a40a3b2f4434779ebf7eb P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:22.855048 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 9890415b2d5a40a3b2f4434779ebf7eb (table rebalance_test_table_12 [id=e0c6bd9b27794a2887eccea514c01058]) successfully deleted
I20240503 13:59:22.855094 9541 tablet_replica.cc:330] T 94960e06ff44463285360fa7f10f1b68 P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:22.855352 9541 raft_consensus.cc:2229] T 94960e06ff44463285360fa7f10f1b68 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:22.855475 9541 raft_consensus.cc:2258] T 94960e06ff44463285360fa7f10f1b68 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:22.855885 9541 ts_tablet_manager.cc:1875] T 94960e06ff44463285360fa7f10f1b68 P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:22.856141 9540 ts_tablet_manager.cc:1888] T 9890415b2d5a40a3b2f4434779ebf7eb P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:22.856216 9540 log.cc:1196] T 9890415b2d5a40a3b2f4434779ebf7eb P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/9890415b2d5a40a3b2f4434779ebf7eb
I20240503 13:59:22.856508 9540 ts_tablet_manager.cc:1909] T 9890415b2d5a40a3b2f4434779ebf7eb P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:22.857144 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet 9890415b2d5a40a3b2f4434779ebf7eb (table rebalance_test_table_12 [id=e0c6bd9b27794a2887eccea514c01058]) successfully deleted
I20240503 13:59:22.857458 9773 raft_consensus.cc:482] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:22.857584 9773 raft_consensus.cc:504] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:22.857852 9541 ts_tablet_manager.cc:1888] T 94960e06ff44463285360fa7f10f1b68 P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:22.858047 9541 log.cc:1196] T 94960e06ff44463285360fa7f10f1b68 P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/94960e06ff44463285360fa7f10f1b68
I20240503 13:59:22.858130 9773 leader_election.cc:288] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:22.858438 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d25b0d68b7643cbbbdeb787cac7a1a4" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:22.858529 9205 raft_consensus.cc:2454] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:22.858613 9541 ts_tablet_manager.cc:1909] T 94960e06ff44463285360fa7f10f1b68 P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:22.858799 9278 leader_election.cc:302] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:22.858968 9773 raft_consensus.cc:2781] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:22.859022 9773 raft_consensus.cc:482] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:22.859076 9773 raft_consensus.cc:3037] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:22.859165 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 94960e06ff44463285360fa7f10f1b68 (table rebalance_test_table_12 [id=e0c6bd9b27794a2887eccea514c01058]) successfully deleted
I20240503 13:59:22.860044 9773 raft_consensus.cc:504] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:22.860404 9773 leader_election.cc:288] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:22.860845 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d25b0d68b7643cbbbdeb787cac7a1a4" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:22.860954 9205 raft_consensus.cc:3037] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:22.861793 9205 raft_consensus.cc:2454] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:22.862068 9278 leader_election.cc:302] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:22.862375 9773 raft_consensus.cc:2781] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:22.862465 9773 raft_consensus.cc:686] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:22.862594 9773 consensus_queue.cc:237] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:22.862908 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d25b0d68b7643cbbbdeb787cac7a1a4" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:22.863058 6790 raft_consensus.cc:2454] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:22.863402 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d25b0d68b7643cbbbdeb787cac7a1a4" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:22.863516 6791 raft_consensus.cc:3037] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:22.864210 6791 raft_consensus.cc:2454] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:22.865367 6383 catalog_manager.cc:5484] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:22.926043 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:22.931363 6639 tablet_service.cc:1467] Processing CreateTablet for tablet 6f7f030a29264d42a42ecce0f75060f8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=676e7f5a91504fd79a79da343b9352f1]), partition=
I20240503 13:59:22.931545 9323 tablet_service.cc:1467] Processing CreateTablet for tablet 6f7f030a29264d42a42ecce0f75060f8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=676e7f5a91504fd79a79da343b9352f1]), partition=
I20240503 13:59:22.931640 6639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f7f030a29264d42a42ecce0f75060f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:22.931747 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f7f030a29264d42a42ecce0f75060f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:22.933682 9768 tablet_bootstrap.cc:492] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:22.934602 9768 tablet_bootstrap.cc:654] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:22.935557 9819 tablet_bootstrap.cc:492] T 6f7f030a29264d42a42ecce0f75060f8 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:22.935788 8927 tablet_service.cc:1467] Processing CreateTablet for tablet 6f7f030a29264d42a42ecce0f75060f8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=676e7f5a91504fd79a79da343b9352f1]), partition=
I20240503 13:59:22.936060 8927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f7f030a29264d42a42ecce0f75060f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:22.936405 9819 tablet_bootstrap.cc:654] T 6f7f030a29264d42a42ecce0f75060f8 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:22.936439 9768 tablet_bootstrap.cc:492] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:22.936506 9768 ts_tablet_manager.cc:1367] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:22.936956 9768 raft_consensus.cc:348] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:22.937036 9768 raft_consensus.cc:374] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:22.937062 9768 raft_consensus.cc:729] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:22.937191 9768 consensus_queue.cc:260] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:22.937487 9768 ts_tablet_manager.cc:1398] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:22.937870 9799 tablet_bootstrap.cc:492] T 6f7f030a29264d42a42ecce0f75060f8 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:22.938139 9819 tablet_bootstrap.cc:492] T 6f7f030a29264d42a42ecce0f75060f8 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:22.938217 9819 ts_tablet_manager.cc:1367] T 6f7f030a29264d42a42ecce0f75060f8 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:22.938295 9773 raft_consensus.cc:482] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:22.938393 9773 raft_consensus.cc:504] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:22.938784 9799 tablet_bootstrap.cc:654] T 6f7f030a29264d42a42ecce0f75060f8 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:22.938796 9773 leader_election.cc:288] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:22.939469 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92eaea6e9c8493690b749f162949ac4" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:22.939631 9205 raft_consensus.cc:2454] T e92eaea6e9c8493690b749f162949ac4 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:22.939880 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92eaea6e9c8493690b749f162949ac4" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:22.940033 8948 raft_consensus.cc:2454] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:22.940282 9278 leader_election.cc:302] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 0d77752181e944698e809e325e5937e6; no voters:
I20240503 13:59:22.940333 9799 tablet_bootstrap.cc:492] T 6f7f030a29264d42a42ecce0f75060f8 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:22.940399 9799 ts_tablet_manager.cc:1367] T 6f7f030a29264d42a42ecce0f75060f8 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:59:22.940414 9819 raft_consensus.cc:348] T 6f7f030a29264d42a42ecce0f75060f8 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:22.940515 9819 raft_consensus.cc:374] T 6f7f030a29264d42a42ecce0f75060f8 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:22.940544 9819 raft_consensus.cc:729] T 6f7f030a29264d42a42ecce0f75060f8 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:22.940575 9773 raft_consensus.cc:2781] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:22.940622 9773 raft_consensus.cc:482] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:22.940656 9773 raft_consensus.cc:3037] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:22.940654 9819 consensus_queue.cc:260] T 6f7f030a29264d42a42ecce0f75060f8 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:22.940845 9799 raft_consensus.cc:348] T 6f7f030a29264d42a42ecce0f75060f8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:22.940925 9799 raft_consensus.cc:374] T 6f7f030a29264d42a42ecce0f75060f8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:22.940953 9799 raft_consensus.cc:729] T 6f7f030a29264d42a42ecce0f75060f8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:22.941012 9819 ts_tablet_manager.cc:1398] T 6f7f030a29264d42a42ecce0f75060f8 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:22.941049 9799 consensus_queue.cc:260] T 6f7f030a29264d42a42ecce0f75060f8 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:22.941314 9799 ts_tablet_manager.cc:1398] T 6f7f030a29264d42a42ecce0f75060f8 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:22.941586 9773 raft_consensus.cc:504] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:22.941905 9773 leader_election.cc:288] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:22.942153 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92eaea6e9c8493690b749f162949ac4" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:22.942236 9205 raft_consensus.cc:3037] T e92eaea6e9c8493690b749f162949ac4 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:22.943092 9205 raft_consensus.cc:2454] T e92eaea6e9c8493690b749f162949ac4 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:22.943318 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92eaea6e9c8493690b749f162949ac4" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:22.943416 8948 raft_consensus.cc:3037] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:22.944238 8948 raft_consensus.cc:2454] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:22.944504 9278 leader_election.cc:302] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 0d77752181e944698e809e325e5937e6; no voters:
I20240503 13:59:22.944654 9773 raft_consensus.cc:2781] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:22.944777 9773 raft_consensus.cc:686] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:22.944903 9773 consensus_queue.cc:237] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:22.946434 6385 catalog_manager.cc:5484] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:23.133221 9773 raft_consensus.cc:482] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:23.133359 9773 raft_consensus.cc:504] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.133792 9773 leader_election.cc:288] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:23.134186 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f7f030a29264d42a42ecce0f75060f8" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:23.134323 8948 raft_consensus.cc:2454] T 6f7f030a29264d42a42ecce0f75060f8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:23.134621 9278 leader_election.cc:302] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 0d77752181e944698e809e325e5937e6; no voters:
I20240503 13:59:23.134799 9773 raft_consensus.cc:2781] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:23.134856 9773 raft_consensus.cc:482] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:23.134904 9773 raft_consensus.cc:3037] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:23.135983 9773 raft_consensus.cc:504] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.136343 9773 leader_election.cc:288] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:23.136656 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f7f030a29264d42a42ecce0f75060f8" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:23.136801 8948 raft_consensus.cc:3037] T 6f7f030a29264d42a42ecce0f75060f8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:23.137785 8948 raft_consensus.cc:2454] T 6f7f030a29264d42a42ecce0f75060f8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:23.138059 9278 leader_election.cc:302] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 0d77752181e944698e809e325e5937e6; no voters:
I20240503 13:59:23.138127 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f7f030a29264d42a42ecce0f75060f8" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:23.138288 6660 raft_consensus.cc:2454] T 6f7f030a29264d42a42ecce0f75060f8 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:23.138473 9773 raft_consensus.cc:2781] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:23.138546 9773 raft_consensus.cc:686] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:23.138680 9773 consensus_queue.cc:237] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.139395 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f7f030a29264d42a42ecce0f75060f8" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:23.139506 6660 raft_consensus.cc:3037] T 6f7f030a29264d42a42ecce0f75060f8 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:23.140223 6660 raft_consensus.cc:2454] T 6f7f030a29264d42a42ecce0f75060f8 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:23.140962 6385 catalog_manager.cc:5484] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:23.272049 6385 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:23.276124 8948 raft_consensus.cc:1261] T 6f7f030a29264d42a42ecce0f75060f8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:23.276185 6660 raft_consensus.cc:1261] T 6f7f030a29264d42a42ecce0f75060f8 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:23.276507 9775 consensus_queue.cc:1035] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:23.276643 9773 consensus_queue.cc:1035] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:23.279899 9834 tablet.cc:1722] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:23.280259 9834 tablet.cc:1620] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:23.280249 6380 catalog_manager.cc:5576] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119) does not have the latest schema for tablet 6f7f030a29264d42a42ecce0f75060f8 (table rebalancer_dynamic_table [id=676e7f5a91504fd79a79da343b9352f1]). Expected version 1 got 0
I20240503 13:59:23.281033 9835 tablet.cc:1722] T 6f7f030a29264d42a42ecce0f75060f8 P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:23.282096 9835 tablet.cc:1620] T 6f7f030a29264d42a42ecce0f75060f8 P 2145d15ab2d74c0287839266f1411a43: MemRowSet was empty: no flush needed.
I20240503 13:59:23.282289 6380 catalog_manager.cc:5576] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931) does not have the latest schema for tablet 6f7f030a29264d42a42ecce0f75060f8 (table rebalancer_dynamic_table [id=676e7f5a91504fd79a79da343b9352f1]). Expected version 1 got 0
I20240503 13:59:23.284045 6385 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet 6f7f030a29264d42a42ecce0f75060f8 (table rebalancer_dynamic_table [id=676e7f5a91504fd79a79da343b9352f1]). Expected version 1 got 0
I20240503 13:59:23.287068 9836 tablet.cc:1722] T 6f7f030a29264d42a42ecce0f75060f8 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:23.287426 9836 tablet.cc:1620] T 6f7f030a29264d42a42ecce0f75060f8 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:23.288340 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=676e7f5a91504fd79a79da343b9352f1] alter complete (version 1)
I20240503 13:59:23.319993 9775 consensus_queue.cc:1035] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:23.323150 9838 consensus_queue.cc:1035] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:23.323634 9828 consensus_queue.cc:1035] T 08c307c79c4b4901be7a4e878c70c4bf P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:23.326856 9762 consensus_queue.cc:1035] T 08c307c79c4b4901be7a4e878c70c4bf P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:23.347596 6385 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240503 13:59:23.347770 6385 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240503 13:59:23.350347 6385 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet b22f2b5852fb48a5a5400f843de82702
I20240503 13:59:23.350749 6385 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 34ede4b3eb8740a9922a061569f6e663
I20240503 13:59:23.350987 6771 tablet_service.cc:1514] Processing DeleteTablet for tablet b22f2b5852fb48a5a5400f843de82702 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:23 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:23.351140 6507 tablet_service.cc:1514] Processing DeleteTablet for tablet b22f2b5852fb48a5a5400f843de82702 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:23 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:23.351353 9731 tablet_replica.cc:330] T b22f2b5852fb48a5a5400f843de82702 P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:23.351352 9541 tablet_replica.cc:330] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:23.351503 9731 raft_consensus.cc:2229] T b22f2b5852fb48a5a5400f843de82702 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:23.351506 9541 raft_consensus.cc:2229] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:23.351521 6771 tablet_service.cc:1514] Processing DeleteTablet for tablet 34ede4b3eb8740a9922a061569f6e663 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:23 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:23.351665 9731 raft_consensus.cc:2258] T b22f2b5852fb48a5a5400f843de82702 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:23.351719 9541 raft_consensus.cc:2258] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:23.351910 6507 tablet_service.cc:1514] Processing DeleteTablet for tablet 34ede4b3eb8740a9922a061569f6e663 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:23 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:23.351990 6639 tablet_service.cc:1514] Processing DeleteTablet for tablet b22f2b5852fb48a5a5400f843de82702 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:23 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:23.352169 9540 tablet_replica.cc:330] T b22f2b5852fb48a5a5400f843de82702 P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:23.352234 9731 ts_tablet_manager.cc:1875] T b22f2b5852fb48a5a5400f843de82702 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:23.352294 6385 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet ba1a0a1ecff746d9ace73f0c2b75d991
I20240503 13:59:23.352296 9541 ts_tablet_manager.cc:1875] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:23.352300 9540 raft_consensus.cc:2229] T b22f2b5852fb48a5a5400f843de82702 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:23.352442 9540 raft_consensus.cc:2258] T b22f2b5852fb48a5a5400f843de82702 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:23.352696 6771 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1a0a1ecff746d9ace73f0c2b75d991 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:23 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:23.352917 9540 ts_tablet_manager.cc:1875] T b22f2b5852fb48a5a5400f843de82702 P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:23.353458 6639 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1a0a1ecff746d9ace73f0c2b75d991 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:23 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:23.354516 9731 ts_tablet_manager.cc:1888] T b22f2b5852fb48a5a5400f843de82702 P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:23.354581 9731 log.cc:1196] T b22f2b5852fb48a5a5400f843de82702 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/b22f2b5852fb48a5a5400f843de82702
I20240503 13:59:23.354612 9541 ts_tablet_manager.cc:1888] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:23.354676 9541 log.cc:1196] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/b22f2b5852fb48a5a5400f843de82702
I20240503 13:59:23.354882 9731 ts_tablet_manager.cc:1909] T b22f2b5852fb48a5a5400f843de82702 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:23.354959 9541 ts_tablet_manager.cc:1909] T b22f2b5852fb48a5a5400f843de82702 P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:23.355376 6901 tablet_service.cc:1514] Processing DeleteTablet for tablet 34ede4b3eb8740a9922a061569f6e663 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:23 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:23.355496 9731 tablet_replica.cc:330] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:23.355592 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet b22f2b5852fb48a5a5400f843de82702 (table rebalance_test_table_14 [id=2593391fce974113886702729b8e3d24]) successfully deleted
I20240503 13:59:23.355616 9731 raft_consensus.cc:2229] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:23.355675 9845 tablet_replica.cc:330] T 34ede4b3eb8740a9922a061569f6e663 P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:23.355782 9731 raft_consensus.cc:2258] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:23.355803 9845 raft_consensus.cc:2229] T 34ede4b3eb8740a9922a061569f6e663 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:23.355947 9845 raft_consensus.cc:2258] T 34ede4b3eb8740a9922a061569f6e663 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:23.356011 6507 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1a0a1ecff746d9ace73f0c2b75d991 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:23 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:23.355269 9540 ts_tablet_manager.cc:1888] T b22f2b5852fb48a5a5400f843de82702 P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:23.356241 9731 ts_tablet_manager.cc:1875] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:23.356278 9540 log.cc:1196] T b22f2b5852fb48a5a5400f843de82702 P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/b22f2b5852fb48a5a5400f843de82702
I20240503 13:59:23.356482 9845 ts_tablet_manager.cc:1875] T 34ede4b3eb8740a9922a061569f6e663 P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:23.356552 9540 ts_tablet_manager.cc:1909] T b22f2b5852fb48a5a5400f843de82702 P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:23.357116 9540 tablet_replica.cc:330] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:23.357230 9540 raft_consensus.cc:2229] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:23.357424 9540 raft_consensus.cc:2258] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:23.357614 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet b22f2b5852fb48a5a5400f843de82702 (table rebalance_test_table_14 [id=2593391fce974113886702729b8e3d24]) successfully deleted
I20240503 13:59:23.357808 9731 ts_tablet_manager.cc:1888] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:23.357869 9731 log.cc:1196] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/34ede4b3eb8740a9922a061569f6e663
I20240503 13:59:23.357909 9540 ts_tablet_manager.cc:1875] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:23.358063 9731 ts_tablet_manager.cc:1909] T 34ede4b3eb8740a9922a061569f6e663 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:23.358716 9845 ts_tablet_manager.cc:1888] T 34ede4b3eb8740a9922a061569f6e663 P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:23.358789 9845 log.cc:1196] T 34ede4b3eb8740a9922a061569f6e663 P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/34ede4b3eb8740a9922a061569f6e663
I20240503 13:59:23.359048 9845 ts_tablet_manager.cc:1909] T 34ede4b3eb8740a9922a061569f6e663 P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:23.359685 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet 34ede4b3eb8740a9922a061569f6e663 (table rebalance_test_table_14 [id=2593391fce974113886702729b8e3d24]) successfully deleted
I20240503 13:59:23.360030 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet b22f2b5852fb48a5a5400f843de82702 (table rebalance_test_table_14 [id=2593391fce974113886702729b8e3d24]) successfully deleted
I20240503 13:59:23.360080 9541 tablet_replica.cc:330] T 34ede4b3eb8740a9922a061569f6e663 P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:23.360245 9541 raft_consensus.cc:2229] T 34ede4b3eb8740a9922a061569f6e663 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:23.360337 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet 34ede4b3eb8740a9922a061569f6e663 (table rebalance_test_table_14 [id=2593391fce974113886702729b8e3d24]) successfully deleted
I20240503 13:59:23.360430 9541 raft_consensus.cc:2258] T 34ede4b3eb8740a9922a061569f6e663 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:23.360483 9731 tablet_replica.cc:330] T ba1a0a1ecff746d9ace73f0c2b75d991 P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:23.360610 9731 raft_consensus.cc:2229] T ba1a0a1ecff746d9ace73f0c2b75d991 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:23.360735 9731 raft_consensus.cc:2258] T ba1a0a1ecff746d9ace73f0c2b75d991 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:23.360886 9541 ts_tablet_manager.cc:1875] T 34ede4b3eb8740a9922a061569f6e663 P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:23.361204 9731 ts_tablet_manager.cc:1875] T ba1a0a1ecff746d9ace73f0c2b75d991 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:23.362399 9541 ts_tablet_manager.cc:1888] T 34ede4b3eb8740a9922a061569f6e663 P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:23.362476 9541 log.cc:1196] T 34ede4b3eb8740a9922a061569f6e663 P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/34ede4b3eb8740a9922a061569f6e663
I20240503 13:59:23.363005 9540 ts_tablet_manager.cc:1888] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:23.363361 9541 ts_tablet_manager.cc:1909] T 34ede4b3eb8740a9922a061569f6e663 P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:23.363381 9731 ts_tablet_manager.cc:1888] T ba1a0a1ecff746d9ace73f0c2b75d991 P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:23.363482 9731 log.cc:1196] T ba1a0a1ecff746d9ace73f0c2b75d991 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ba1a0a1ecff746d9ace73f0c2b75d991
I20240503 13:59:23.363484 9540 log.cc:1196] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/ba1a0a1ecff746d9ace73f0c2b75d991
I20240503 13:59:23.363765 9540 ts_tablet_manager.cc:1909] T ba1a0a1ecff746d9ace73f0c2b75d991 P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:23.363870 9731 ts_tablet_manager.cc:1909] T ba1a0a1ecff746d9ace73f0c2b75d991 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:23.364367 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet ba1a0a1ecff746d9ace73f0c2b75d991 (table rebalance_test_table_14 [id=2593391fce974113886702729b8e3d24]) successfully deleted
I20240503 13:59:23.364739 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet ba1a0a1ecff746d9ace73f0c2b75d991 (table rebalance_test_table_14 [id=2593391fce974113886702729b8e3d24]) successfully deleted
I20240503 13:59:23.364894 9541 tablet_replica.cc:330] T ba1a0a1ecff746d9ace73f0c2b75d991 P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:23.364934 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 34ede4b3eb8740a9922a061569f6e663 (table rebalance_test_table_14 [id=2593391fce974113886702729b8e3d24]) successfully deleted
I20240503 13:59:23.365026 9541 raft_consensus.cc:2229] T ba1a0a1ecff746d9ace73f0c2b75d991 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:23.365176 9541 raft_consensus.cc:2258] T ba1a0a1ecff746d9ace73f0c2b75d991 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:23.365605 9541 ts_tablet_manager.cc:1875] T ba1a0a1ecff746d9ace73f0c2b75d991 P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:23.367403 9541 ts_tablet_manager.cc:1888] T ba1a0a1ecff746d9ace73f0c2b75d991 P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:23.367525 9541 log.cc:1196] T ba1a0a1ecff746d9ace73f0c2b75d991 P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ba1a0a1ecff746d9ace73f0c2b75d991
I20240503 13:59:23.367759 9541 ts_tablet_manager.cc:1909] T ba1a0a1ecff746d9ace73f0c2b75d991 P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:23.368487 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet ba1a0a1ecff746d9ace73f0c2b75d991 (table rebalance_test_table_14 [id=2593391fce974113886702729b8e3d24]) successfully deleted
I20240503 13:59:23.392864 6385 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:23.397554 9834 tablet.cc:1722] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:23.397887 9834 tablet.cc:1620] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:23.398114 9836 tablet.cc:1722] T 6f7f030a29264d42a42ecce0f75060f8 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:23.398414 9836 tablet.cc:1620] T 6f7f030a29264d42a42ecce0f75060f8 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:23.400753 9835 tablet.cc:1722] T 6f7f030a29264d42a42ecce0f75060f8 P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:23.401023 9835 tablet.cc:1620] T 6f7f030a29264d42a42ecce0f75060f8 P 2145d15ab2d74c0287839266f1411a43: MemRowSet was empty: no flush needed.
I20240503 13:59:23.402868 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=676e7f5a91504fd79a79da343b9352f1] alter complete (version 2)
I20240503 13:59:23.469798 9773 consensus_queue.cc:1035] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:23.472576 9838 consensus_queue.cc:1035] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:23.489392 6380 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:23.494724 6507 tablet_service.cc:1467] Processing CreateTablet for tablet f853508f20d4431bb35e3e05b659953a (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=b511123c500546da8d5ca21cb43ec824]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:23.494747 9190 tablet_service.cc:1467] Processing CreateTablet for tablet f853508f20d4431bb35e3e05b659953a (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=b511123c500546da8d5ca21cb43ec824]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:23.495031 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f853508f20d4431bb35e3e05b659953a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:23.495031 6507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f853508f20d4431bb35e3e05b659953a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:23.495627 9191 tablet_service.cc:1467] Processing CreateTablet for tablet 038537d00e904bca89f0214d04fb4f13 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=b511123c500546da8d5ca21cb43ec824]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:23.495841 9191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 038537d00e904bca89f0214d04fb4f13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:23.495750 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 9253d1943fd74523bafe01c1acf79188 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=b511123c500546da8d5ca21cb43ec824]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:23.496047 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9253d1943fd74523bafe01c1acf79188. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:23.496135 6771 tablet_service.cc:1467] Processing CreateTablet for tablet 9253d1943fd74523bafe01c1acf79188 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=b511123c500546da8d5ca21cb43ec824]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:23.496381 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9253d1943fd74523bafe01c1acf79188. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:23.496320 9061 tablet_service.cc:1467] Processing CreateTablet for tablet 038537d00e904bca89f0214d04fb4f13 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=b511123c500546da8d5ca21cb43ec824]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:23.496565 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 038537d00e904bca89f0214d04fb4f13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:23.498205 9849 tablet_bootstrap.cc:492] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:23.498899 9848 tablet_bootstrap.cc:492] T f853508f20d4431bb35e3e05b659953a P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:23.499070 8927 tablet_service.cc:1467] Processing CreateTablet for tablet f853508f20d4431bb35e3e05b659953a (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=b511123c500546da8d5ca21cb43ec824]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:23.499157 9849 tablet_bootstrap.cc:654] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:23.499379 8927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f853508f20d4431bb35e3e05b659953a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:23.499845 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 038537d00e904bca89f0214d04fb4f13 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=b511123c500546da8d5ca21cb43ec824]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:23.499900 9848 tablet_bootstrap.cc:654] T f853508f20d4431bb35e3e05b659953a P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:23.500022 9850 tablet_bootstrap.cc:492] T 9253d1943fd74523bafe01c1acf79188 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:23.500051 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 038537d00e904bca89f0214d04fb4f13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:23.500895 9323 tablet_service.cc:1467] Processing CreateTablet for tablet 9253d1943fd74523bafe01c1acf79188 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=b511123c500546da8d5ca21cb43ec824]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:23.500985 9850 tablet_bootstrap.cc:654] T 9253d1943fd74523bafe01c1acf79188 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:23.501073 9848 tablet_bootstrap.cc:492] T f853508f20d4431bb35e3e05b659953a P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:23.501158 9849 tablet_bootstrap.cc:492] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:23.501170 9848 ts_tablet_manager.cc:1367] T f853508f20d4431bb35e3e05b659953a P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:23.501189 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9253d1943fd74523bafe01c1acf79188. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:23.501266 9849 ts_tablet_manager.cc:1367] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:59:23.501571 9848 raft_consensus.cc:348] T f853508f20d4431bb35e3e05b659953a P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.501667 9848 raft_consensus.cc:374] T f853508f20d4431bb35e3e05b659953a P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:23.501694 9848 raft_consensus.cc:729] T f853508f20d4431bb35e3e05b659953a P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:23.501719 9849 raft_consensus.cc:348] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:23.501843 9849 raft_consensus.cc:374] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:23.501813 9848 consensus_queue.cc:260] T f853508f20d4431bb35e3e05b659953a P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.501875 9849 raft_consensus.cc:729] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:23.501977 9849 consensus_queue.cc:260] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:23.502107 9848 ts_tablet_manager.cc:1398] T f853508f20d4431bb35e3e05b659953a P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:23.502240 9849 ts_tablet_manager.cc:1398] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:23.502394 9849 tablet_bootstrap.cc:492] T 038537d00e904bca89f0214d04fb4f13 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:23.503258 9849 tablet_bootstrap.cc:654] T 038537d00e904bca89f0214d04fb4f13 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:23.503255 9854 tablet_bootstrap.cc:492] T 9253d1943fd74523bafe01c1acf79188 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:23.504190 9854 tablet_bootstrap.cc:654] T 9253d1943fd74523bafe01c1acf79188 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:23.505017 9850 tablet_bootstrap.cc:492] T 9253d1943fd74523bafe01c1acf79188 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:23.505113 9850 ts_tablet_manager.cc:1367] T 9253d1943fd74523bafe01c1acf79188 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:59:23.505364 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:23.505568 9850 raft_consensus.cc:348] T 9253d1943fd74523bafe01c1acf79188 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:23.505678 9850 raft_consensus.cc:374] T 9253d1943fd74523bafe01c1acf79188 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:23.505718 9850 raft_consensus.cc:729] T 9253d1943fd74523bafe01c1acf79188 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:23.505856 9850 consensus_queue.cc:260] T 9253d1943fd74523bafe01c1acf79188 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:23.506762 9850 ts_tablet_manager.cc:1398] T 9253d1943fd74523bafe01c1acf79188 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:23.507104 9854 tablet_bootstrap.cc:492] T 9253d1943fd74523bafe01c1acf79188 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:23.507193 9854 ts_tablet_manager.cc:1367] T 9253d1943fd74523bafe01c1acf79188 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 13:59:23.507480 9849 tablet_bootstrap.cc:492] T 038537d00e904bca89f0214d04fb4f13 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:23.507575 9849 ts_tablet_manager.cc:1367] T 038537d00e904bca89f0214d04fb4f13 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:23.507898 9854 raft_consensus.cc:348] T 9253d1943fd74523bafe01c1acf79188 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:23.507990 9854 raft_consensus.cc:374] T 9253d1943fd74523bafe01c1acf79188 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:23.508045 9854 raft_consensus.cc:729] T 9253d1943fd74523bafe01c1acf79188 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:23.508051 9849 raft_consensus.cc:348] T 038537d00e904bca89f0214d04fb4f13 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.508160 9849 raft_consensus.cc:374] T 038537d00e904bca89f0214d04fb4f13 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:23.508200 9849 raft_consensus.cc:729] T 038537d00e904bca89f0214d04fb4f13 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:23.508205 9854 consensus_queue.cc:260] T 9253d1943fd74523bafe01c1acf79188 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:23.508316 9849 consensus_queue.cc:260] T 038537d00e904bca89f0214d04fb4f13 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.508576 9849 ts_tablet_manager.cc:1398] T 038537d00e904bca89f0214d04fb4f13 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:23.508723 9854 ts_tablet_manager.cc:1398] T 9253d1943fd74523bafe01c1acf79188 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:23.513433 9857 tablet_bootstrap.cc:492] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:23.513554 9836 tablet.cc:1722] T 6f7f030a29264d42a42ecce0f75060f8 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:23.513922 9835 tablet.cc:1722] T 6f7f030a29264d42a42ecce0f75060f8 P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:23.514529 9857 tablet_bootstrap.cc:654] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:23.514873 9834 tablet.cc:1722] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:23.515643 9857 tablet_bootstrap.cc:492] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:23.515756 9857 ts_tablet_manager.cc:1367] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:23.516302 9857 raft_consensus.cc:348] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.516484 9857 raft_consensus.cc:374] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:23.516562 9857 raft_consensus.cc:729] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:23.516768 9857 consensus_queue.cc:260] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.517184 9857 ts_tablet_manager.cc:1398] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:23.517380 9857 tablet_bootstrap.cc:492] T 038537d00e904bca89f0214d04fb4f13 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:23.521445 9857 tablet_bootstrap.cc:654] T 038537d00e904bca89f0214d04fb4f13 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:23.522150 9859 tablet_bootstrap.cc:492] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:23.522519 9857 tablet_bootstrap.cc:492] T 038537d00e904bca89f0214d04fb4f13 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:23.522596 9857 ts_tablet_manager.cc:1367] T 038537d00e904bca89f0214d04fb4f13 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:23.523257 9859 tablet_bootstrap.cc:654] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:23.523200 9857 raft_consensus.cc:348] T 038537d00e904bca89f0214d04fb4f13 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.523344 9857 raft_consensus.cc:374] T 038537d00e904bca89f0214d04fb4f13 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:23.523380 9857 raft_consensus.cc:729] T 038537d00e904bca89f0214d04fb4f13 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:23.523512 9857 consensus_queue.cc:260] T 038537d00e904bca89f0214d04fb4f13 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.523789 9857 ts_tablet_manager.cc:1398] T 038537d00e904bca89f0214d04fb4f13 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:23.523700 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=676e7f5a91504fd79a79da343b9352f1] alter complete (version 3)
I20240503 13:59:23.525271 9859 tablet_bootstrap.cc:492] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:23.525373 9859 ts_tablet_manager.cc:1367] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:23.525846 9859 raft_consensus.cc:348] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.525957 9859 raft_consensus.cc:374] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:23.525990 9859 raft_consensus.cc:729] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:23.526130 9859 consensus_queue.cc:260] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.526499 9859 ts_tablet_manager.cc:1398] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:23.526654 9859 tablet_bootstrap.cc:492] T f853508f20d4431bb35e3e05b659953a P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:23.527631 9859 tablet_bootstrap.cc:654] T f853508f20d4431bb35e3e05b659953a P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:23.528657 9859 tablet_bootstrap.cc:492] T f853508f20d4431bb35e3e05b659953a P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:23.528743 9859 ts_tablet_manager.cc:1367] T f853508f20d4431bb35e3e05b659953a P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:23.529349 9859 raft_consensus.cc:348] T f853508f20d4431bb35e3e05b659953a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.529430 9859 raft_consensus.cc:374] T f853508f20d4431bb35e3e05b659953a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:23.529462 9859 raft_consensus.cc:729] T f853508f20d4431bb35e3e05b659953a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:23.529609 9859 consensus_queue.cc:260] T f853508f20d4431bb35e3e05b659953a P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.529897 9859 ts_tablet_manager.cc:1398] T f853508f20d4431bb35e3e05b659953a P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:23.573585 9725 raft_consensus.cc:482] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:23.573709 9725 raft_consensus.cc:504] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:23.574223 9725 leader_election.cc:288] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:23.574790 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9253d1943fd74523bafe01c1acf79188" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:23.575006 9343 raft_consensus.cc:2454] T 9253d1943fd74523bafe01c1acf79188 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:23.575387 9017 leader_election.cc:302] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:23.575534 9725 raft_consensus.cc:2781] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:23.575583 9725 raft_consensus.cc:482] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:23.575611 9725 raft_consensus.cc:3037] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:23.576601 9725 raft_consensus.cc:504] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:23.576936 9725 leader_election.cc:288] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:23.577379 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9253d1943fd74523bafe01c1acf79188" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:23.577494 9343 raft_consensus.cc:3037] T 9253d1943fd74523bafe01c1acf79188 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:23.578262 9343 raft_consensus.cc:2454] T 9253d1943fd74523bafe01c1acf79188 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:23.578528 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9253d1943fd74523bafe01c1acf79188" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:23.578578 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9253d1943fd74523bafe01c1acf79188" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:23.578672 6790 raft_consensus.cc:2454] T 9253d1943fd74523bafe01c1acf79188 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:23.578872 9017 leader_election.cc:302] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:23.579016 9725 raft_consensus.cc:2781] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:23.579110 9725 raft_consensus.cc:686] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 13:59:23.579245 9725 consensus_queue.cc:237] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:23.580947 6380 catalog_manager.cc:5484] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:23.628343 6380 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:23.628490 6380 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:23.630358 6380 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 6f7f030a29264d42a42ecce0f75060f8
I20240503 13:59:23.630867 6639 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f7f030a29264d42a42ecce0f75060f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:23 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:23.631012 9323 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f7f030a29264d42a42ecce0f75060f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:23 UTC) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:23.631325 9540 tablet_replica.cc:330] T 6f7f030a29264d42a42ecce0f75060f8 P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:23.631325 8927 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f7f030a29264d42a42ecce0f75060f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:23 UTC) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:23.631383 9866 tablet_replica.cc:330] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:23.631474 9540 raft_consensus.cc:2229] T 6f7f030a29264d42a42ecce0f75060f8 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:23.631489 9866 raft_consensus.cc:2229] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:23.631582 9540 raft_consensus.cc:2258] T 6f7f030a29264d42a42ecce0f75060f8 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:23.631691 9866 raft_consensus.cc:2258] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:23.632328 9866 ts_tablet_manager.cc:1875] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:23.632328 9540 ts_tablet_manager.cc:1875] T 6f7f030a29264d42a42ecce0f75060f8 P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:23.633702 9867 tablet_replica.cc:330] T 6f7f030a29264d42a42ecce0f75060f8 P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:23.633877 9867 raft_consensus.cc:2229] T 6f7f030a29264d42a42ecce0f75060f8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:23.634050 9867 raft_consensus.cc:2258] T 6f7f030a29264d42a42ecce0f75060f8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:23.634672 9540 ts_tablet_manager.cc:1888] T 6f7f030a29264d42a42ecce0f75060f8 P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:23.634688 9867 ts_tablet_manager.cc:1875] T 6f7f030a29264d42a42ecce0f75060f8 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:23.634793 9540 log.cc:1196] T 6f7f030a29264d42a42ecce0f75060f8 P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/6f7f030a29264d42a42ecce0f75060f8
I20240503 13:59:23.635071 9540 ts_tablet_manager.cc:1909] T 6f7f030a29264d42a42ecce0f75060f8 P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:23.635519 9866 ts_tablet_manager.cc:1888] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:23.635651 9866 log.cc:1196] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/6f7f030a29264d42a42ecce0f75060f8
I20240503 13:59:23.635795 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet 6f7f030a29264d42a42ecce0f75060f8 (table rebalancer_dynamic_table_new_name [id=676e7f5a91504fd79a79da343b9352f1]) successfully deleted
I20240503 13:59:23.636170 9866 ts_tablet_manager.cc:1909] T 6f7f030a29264d42a42ecce0f75060f8 P 0d77752181e944698e809e325e5937e6: Deleting consensus metadata
I20240503 13:59:23.636718 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet 6f7f030a29264d42a42ecce0f75060f8 (table rebalancer_dynamic_table_new_name [id=676e7f5a91504fd79a79da343b9352f1]) successfully deleted
I20240503 13:59:23.636761 9867 ts_tablet_manager.cc:1888] T 6f7f030a29264d42a42ecce0f75060f8 P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:23.636821 9867 log.cc:1196] T 6f7f030a29264d42a42ecce0f75060f8 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/6f7f030a29264d42a42ecce0f75060f8
I20240503 13:59:23.637073 9867 ts_tablet_manager.cc:1909] T 6f7f030a29264d42a42ecce0f75060f8 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting consensus metadata
I20240503 13:59:23.637673 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 6f7f030a29264d42a42ecce0f75060f8 (table rebalancer_dynamic_table_new_name [id=676e7f5a91504fd79a79da343b9352f1]) successfully deleted
I20240503 13:59:23.638779 9717 raft_consensus.cc:482] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:23.638861 9717 raft_consensus.cc:504] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.639197 9717 leader_election.cc:288] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:23.639591 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038537d00e904bca89f0214d04fb4f13" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:23.639632 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038537d00e904bca89f0214d04fb4f13" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:23.639729 9081 raft_consensus.cc:2454] T 038537d00e904bca89f0214d04fb4f13 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:23.639756 9205 raft_consensus.cc:2454] T 038537d00e904bca89f0214d04fb4f13 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:23.640043 8883 leader_election.cc:302] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:23.640208 9717 raft_consensus.cc:2781] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:23.640264 9717 raft_consensus.cc:482] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:23.640311 9717 raft_consensus.cc:3037] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:23.640944 9717 raft_consensus.cc:504] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.641199 9717 leader_election.cc:288] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:23.641558 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038537d00e904bca89f0214d04fb4f13" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:23.641669 9205 raft_consensus.cc:3037] T 038537d00e904bca89f0214d04fb4f13 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:23.642378 9205 raft_consensus.cc:2454] T 038537d00e904bca89f0214d04fb4f13 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:23.642634 8883 leader_election.cc:302] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:23.642741 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038537d00e904bca89f0214d04fb4f13" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:23.642794 9717 raft_consensus.cc:2781] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:23.642854 9081 raft_consensus.cc:3037] T 038537d00e904bca89f0214d04fb4f13 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:23.642889 9717 raft_consensus.cc:686] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 13:59:23.642997 9717 consensus_queue.cc:237] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.643705 9081 raft_consensus.cc:2454] T 038537d00e904bca89f0214d04fb4f13 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:23.644614 6380 catalog_manager.cc:5484] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:23.661164 9762 raft_consensus.cc:482] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:23.661266 9762 raft_consensus.cc:504] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.661641 9762 leader_election.cc:288] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:23.661971 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f853508f20d4431bb35e3e05b659953a" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:23.661994 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f853508f20d4431bb35e3e05b659953a" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:23.662111 8948 raft_consensus.cc:2454] T f853508f20d4431bb35e3e05b659953a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:23.662111 6528 raft_consensus.cc:2454] T f853508f20d4431bb35e3e05b659953a P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:23.662381 9147 leader_election.cc:302] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:23.662541 9762 raft_consensus.cc:2781] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:23.662600 9762 raft_consensus.cc:482] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:23.662642 9762 raft_consensus.cc:3037] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:23.663549 9762 raft_consensus.cc:504] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.663821 9762 leader_election.cc:288] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:23.664103 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f853508f20d4431bb35e3e05b659953a" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:23.664196 6528 raft_consensus.cc:3037] T f853508f20d4431bb35e3e05b659953a P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:23.664165 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f853508f20d4431bb35e3e05b659953a" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:23.664275 8948 raft_consensus.cc:3037] T f853508f20d4431bb35e3e05b659953a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:23.664997 6528 raft_consensus.cc:2454] T f853508f20d4431bb35e3e05b659953a P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:23.665051 8948 raft_consensus.cc:2454] T f853508f20d4431bb35e3e05b659953a P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:23.665246 9147 leader_election.cc:302] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:23.665424 9762 raft_consensus.cc:2781] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:23.665516 9762 raft_consensus.cc:686] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 13:59:23.665639 9762 consensus_queue.cc:237] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.667110 6380 catalog_manager.cc:5484] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:23.731917 6380 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:23.736929 9191 tablet_service.cc:1467] Processing CreateTablet for tablet 04c63a06c2e243b9a931b951572da020 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3263b563e6764dcd9d00e2efbd89fc48]), partition=
I20240503 13:59:23.737030 9061 tablet_service.cc:1467] Processing CreateTablet for tablet 04c63a06c2e243b9a931b951572da020 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3263b563e6764dcd9d00e2efbd89fc48]), partition=
I20240503 13:59:23.737211 9191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04c63a06c2e243b9a931b951572da020. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:23.737264 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04c63a06c2e243b9a931b951572da020. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:23.738929 9857 tablet_bootstrap.cc:492] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:23.739313 9849 tablet_bootstrap.cc:492] T 04c63a06c2e243b9a931b951572da020 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:23.740059 9857 tablet_bootstrap.cc:654] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:23.740166 9849 tablet_bootstrap.cc:654] T 04c63a06c2e243b9a931b951572da020 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:23.740866 8927 tablet_service.cc:1467] Processing CreateTablet for tablet 04c63a06c2e243b9a931b951572da020 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3263b563e6764dcd9d00e2efbd89fc48]), partition=
I20240503 13:59:23.741118 8927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04c63a06c2e243b9a931b951572da020. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:23.742573 9849 tablet_bootstrap.cc:492] T 04c63a06c2e243b9a931b951572da020 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:23.742641 9849 ts_tablet_manager.cc:1367] T 04c63a06c2e243b9a931b951572da020 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:23.743053 9859 tablet_bootstrap.cc:492] T 04c63a06c2e243b9a931b951572da020 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:23.743053 9849 raft_consensus.cc:348] T 04c63a06c2e243b9a931b951572da020 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.743144 9849 raft_consensus.cc:374] T 04c63a06c2e243b9a931b951572da020 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:23.743172 9849 raft_consensus.cc:729] T 04c63a06c2e243b9a931b951572da020 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:23.743273 9849 consensus_queue.cc:260] T 04c63a06c2e243b9a931b951572da020 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.743566 9849 ts_tablet_manager.cc:1398] T 04c63a06c2e243b9a931b951572da020 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 13:59:23.743981 9859 tablet_bootstrap.cc:654] T 04c63a06c2e243b9a931b951572da020 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:23.744767 9857 tablet_bootstrap.cc:492] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:23.744913 9857 ts_tablet_manager.cc:1367] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:59:23.745386 9857 raft_consensus.cc:348] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.745496 9857 raft_consensus.cc:374] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:23.745529 9857 raft_consensus.cc:729] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:23.745641 9857 consensus_queue.cc:260] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.745939 9857 ts_tablet_manager.cc:1398] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:23.748028 9859 tablet_bootstrap.cc:492] T 04c63a06c2e243b9a931b951572da020 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:23.748114 9859 ts_tablet_manager.cc:1367] T 04c63a06c2e243b9a931b951572da020 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:23.748579 9859 raft_consensus.cc:348] T 04c63a06c2e243b9a931b951572da020 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.748692 9859 raft_consensus.cc:374] T 04c63a06c2e243b9a931b951572da020 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:23.748734 9859 raft_consensus.cc:729] T 04c63a06c2e243b9a931b951572da020 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:23.748883 9859 consensus_queue.cc:260] T 04c63a06c2e243b9a931b951572da020 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.749171 9859 ts_tablet_manager.cc:1398] T 04c63a06c2e243b9a931b951572da020 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:23.752315 9762 raft_consensus.cc:482] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:23.752396 9762 raft_consensus.cc:504] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.752736 9762 leader_election.cc:288] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:23.753036 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04c63a06c2e243b9a931b951572da020" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:23.753158 9081 raft_consensus.cc:2454] T 04c63a06c2e243b9a931b951572da020 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:23.753366 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04c63a06c2e243b9a931b951572da020" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:23.753424 9147 leader_election.cc:302] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:23.753472 8948 raft_consensus.cc:2454] T 04c63a06c2e243b9a931b951572da020 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:23.753636 9762 raft_consensus.cc:2781] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:23.753685 9762 raft_consensus.cc:482] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:23.753731 9762 raft_consensus.cc:3037] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:23.754523 9762 raft_consensus.cc:504] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.754765 9762 leader_election.cc:288] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:23.755118 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04c63a06c2e243b9a931b951572da020" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:23.755230 8948 raft_consensus.cc:3037] T 04c63a06c2e243b9a931b951572da020 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:23.755209 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04c63a06c2e243b9a931b951572da020" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:23.755326 9081 raft_consensus.cc:3037] T 04c63a06c2e243b9a931b951572da020 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:23.756098 8948 raft_consensus.cc:2454] T 04c63a06c2e243b9a931b951572da020 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:23.756176 9081 raft_consensus.cc:2454] T 04c63a06c2e243b9a931b951572da020 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:23.756386 9147 leader_election.cc:302] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:23.756649 9762 raft_consensus.cc:2781] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:23.756739 9762 raft_consensus.cc:686] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 13:59:23.756865 9762 consensus_queue.cc:237] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:23.758204 6379 catalog_manager.cc:5484] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:23.856909 6379 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240503 13:59:23.857084 6379 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240503 13:59:23.859468 6379 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet e5588e9e339a498a984bee413013bdce
I20240503 13:59:23.859777 6379 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet b56aa48ef99c4cf89f4147c4f863005b
I20240503 13:59:23.860141 6901 tablet_service.cc:1514] Processing DeleteTablet for tablet e5588e9e339a498a984bee413013bdce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:23 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:23.860242 6379 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 1c324cc92ff744a9a9283f64b471e347
I20240503 13:59:23.860335 6639 tablet_service.cc:1514] Processing DeleteTablet for tablet e5588e9e339a498a984bee413013bdce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:23 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:23.860421 6640 tablet_service.cc:1514] Processing DeleteTablet for tablet b56aa48ef99c4cf89f4147c4f863005b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:23 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:23.860553 6507 tablet_service.cc:1514] Processing DeleteTablet for tablet e5588e9e339a498a984bee413013bdce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:23 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:23.860589 9540 tablet_replica.cc:330] T e5588e9e339a498a984bee413013bdce P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:23.860622 6508 tablet_service.cc:1514] Processing DeleteTablet for tablet b56aa48ef99c4cf89f4147c4f863005b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:23 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:23.860716 9540 raft_consensus.cc:2229] T e5588e9e339a498a984bee413013bdce P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:23.860852 9540 raft_consensus.cc:2258] T e5588e9e339a498a984bee413013bdce P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:23.860985 6769 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c324cc92ff744a9a9283f64b471e347 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:23 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:23.861101 9541 tablet_replica.cc:330] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:23.861172 9731 tablet_replica.cc:330] T 1c324cc92ff744a9a9283f64b471e347 P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:23.861240 9541 raft_consensus.cc:2229] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:23.861315 6901 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c324cc92ff744a9a9283f64b471e347 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:23 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:23.861322 9731 raft_consensus.cc:2229] T 1c324cc92ff744a9a9283f64b471e347 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:23.861443 9731 raft_consensus.cc:2258] T 1c324cc92ff744a9a9283f64b471e347 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:23.861449 9541 raft_consensus.cc:2258] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:23.861894 9875 tablet_replica.cc:330] T e5588e9e339a498a984bee413013bdce P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:23.861924 9731 ts_tablet_manager.cc:1875] T 1c324cc92ff744a9a9283f64b471e347 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:23.861963 9540 ts_tablet_manager.cc:1875] T e5588e9e339a498a984bee413013bdce P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:23.861974 9541 ts_tablet_manager.cc:1875] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:23.862054 9875 raft_consensus.cc:2229] T e5588e9e339a498a984bee413013bdce P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:23.862175 9875 raft_consensus.cc:2258] T e5588e9e339a498a984bee413013bdce P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:23.862692 9875 ts_tablet_manager.cc:1875] T e5588e9e339a498a984bee413013bdce P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:23.864194 9541 ts_tablet_manager.cc:1888] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:23.864264 9541 log.cc:1196] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/e5588e9e339a498a984bee413013bdce
I20240503 13:59:23.864248 9731 ts_tablet_manager.cc:1888] T 1c324cc92ff744a9a9283f64b471e347 P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:23.864315 9731 log.cc:1196] T 1c324cc92ff744a9a9283f64b471e347 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/1c324cc92ff744a9a9283f64b471e347
I20240503 13:59:23.864339 9540 ts_tablet_manager.cc:1888] T e5588e9e339a498a984bee413013bdce P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:23.864399 9540 log.cc:1196] T e5588e9e339a498a984bee413013bdce P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/e5588e9e339a498a984bee413013bdce
I20240503 13:59:23.864542 9541 ts_tablet_manager.cc:1909] T e5588e9e339a498a984bee413013bdce P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:23.864637 9731 ts_tablet_manager.cc:1909] T 1c324cc92ff744a9a9283f64b471e347 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:23.864651 9540 ts_tablet_manager.cc:1909] T e5588e9e339a498a984bee413013bdce P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:23.864734 6771 tablet_service.cc:1514] Processing DeleteTablet for tablet b56aa48ef99c4cf89f4147c4f863005b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:23 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:23.864910 6508 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c324cc92ff744a9a9283f64b471e347 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:23 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:23.865259 9540 tablet_replica.cc:330] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:23.865401 9540 raft_consensus.cc:2229] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:23.865608 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet e5588e9e339a498a984bee413013bdce (table rebalance_test_table_16 [id=c7a03df129af49a4a7f1052b1e4a915c]) successfully deleted
I20240503 13:59:23.865725 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet e5588e9e339a498a984bee413013bdce (table rebalance_test_table_16 [id=c7a03df129af49a4a7f1052b1e4a915c]) successfully deleted
I20240503 13:59:23.865859 9540 raft_consensus.cc:2258] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:23.865898 9875 ts_tablet_manager.cc:1888] T e5588e9e339a498a984bee413013bdce P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:23.865939 9541 tablet_replica.cc:330] T b56aa48ef99c4cf89f4147c4f863005b P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:23.866024 9875 log.cc:1196] T e5588e9e339a498a984bee413013bdce P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/e5588e9e339a498a984bee413013bdce
I20240503 13:59:23.866065 9541 raft_consensus.cc:2229] T b56aa48ef99c4cf89f4147c4f863005b P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:23.866143 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet 1c324cc92ff744a9a9283f64b471e347 (table rebalance_test_table_16 [id=c7a03df129af49a4a7f1052b1e4a915c]) successfully deleted
I20240503 13:59:23.866227 9541 raft_consensus.cc:2258] T b56aa48ef99c4cf89f4147c4f863005b P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:23.866324 9875 ts_tablet_manager.cc:1909] T e5588e9e339a498a984bee413013bdce P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:23.866367 9540 ts_tablet_manager.cc:1875] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:23.866467 9731 tablet_replica.cc:330] T b56aa48ef99c4cf89f4147c4f863005b P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:23.866639 9731 raft_consensus.cc:2229] T b56aa48ef99c4cf89f4147c4f863005b P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:23.866690 9541 ts_tablet_manager.cc:1875] T b56aa48ef99c4cf89f4147c4f863005b P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:23.866804 9731 raft_consensus.cc:2258] T b56aa48ef99c4cf89f4147c4f863005b P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:23.866930 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet e5588e9e339a498a984bee413013bdce (table rebalance_test_table_16 [id=c7a03df129af49a4a7f1052b1e4a915c]) successfully deleted
I20240503 13:59:23.867005 9875 tablet_replica.cc:330] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:23.867156 9875 raft_consensus.cc:2229] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:23.867370 9875 raft_consensus.cc:2258] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:23.867473 9731 ts_tablet_manager.cc:1875] T b56aa48ef99c4cf89f4147c4f863005b P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:23.867939 9875 ts_tablet_manager.cc:1875] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:23.868436 9540 ts_tablet_manager.cc:1888] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:23.868552 9540 log.cc:1196] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/b56aa48ef99c4cf89f4147c4f863005b
I20240503 13:59:23.868587 9541 ts_tablet_manager.cc:1888] T b56aa48ef99c4cf89f4147c4f863005b P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:23.868698 9541 log.cc:1196] T b56aa48ef99c4cf89f4147c4f863005b P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/b56aa48ef99c4cf89f4147c4f863005b
I20240503 13:59:23.868850 9540 ts_tablet_manager.cc:1909] T b56aa48ef99c4cf89f4147c4f863005b P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:23.868994 9541 ts_tablet_manager.cc:1909] T b56aa48ef99c4cf89f4147c4f863005b P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:23.869458 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet b56aa48ef99c4cf89f4147c4f863005b (table rebalance_test_table_16 [id=c7a03df129af49a4a7f1052b1e4a915c]) successfully deleted
I20240503 13:59:23.869685 9731 ts_tablet_manager.cc:1888] T b56aa48ef99c4cf89f4147c4f863005b P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:23.869745 9731 log.cc:1196] T b56aa48ef99c4cf89f4147c4f863005b P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/b56aa48ef99c4cf89f4147c4f863005b
I20240503 13:59:23.869872 9875 ts_tablet_manager.cc:1888] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:23.869874 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet b56aa48ef99c4cf89f4147c4f863005b (table rebalance_test_table_16 [id=c7a03df129af49a4a7f1052b1e4a915c]) successfully deleted
I20240503 13:59:23.869911 9541 tablet_replica.cc:330] T 1c324cc92ff744a9a9283f64b471e347 P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:23.869990 9875 log.cc:1196] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/1c324cc92ff744a9a9283f64b471e347
I20240503 13:59:23.870020 9731 ts_tablet_manager.cc:1909] T b56aa48ef99c4cf89f4147c4f863005b P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:23.870025 9541 raft_consensus.cc:2229] T 1c324cc92ff744a9a9283f64b471e347 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:23.870146 9541 raft_consensus.cc:2258] T 1c324cc92ff744a9a9283f64b471e347 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:23.870268 9875 ts_tablet_manager.cc:1909] T 1c324cc92ff744a9a9283f64b471e347 P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:23.870631 9541 ts_tablet_manager.cc:1875] T 1c324cc92ff744a9a9283f64b471e347 P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:23.870635 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet b56aa48ef99c4cf89f4147c4f863005b (table rebalance_test_table_16 [id=c7a03df129af49a4a7f1052b1e4a915c]) successfully deleted
I20240503 13:59:23.870833 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet 1c324cc92ff744a9a9283f64b471e347 (table rebalance_test_table_16 [id=c7a03df129af49a4a7f1052b1e4a915c]) successfully deleted
I20240503 13:59:23.870841 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:23.872381 9541 ts_tablet_manager.cc:1888] T 1c324cc92ff744a9a9283f64b471e347 P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:23.872462 9541 log.cc:1196] T 1c324cc92ff744a9a9283f64b471e347 P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/1c324cc92ff744a9a9283f64b471e347
I20240503 13:59:23.872712 9541 ts_tablet_manager.cc:1909] T 1c324cc92ff744a9a9283f64b471e347 P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:23.873384 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 1c324cc92ff744a9a9283f64b471e347 (table rebalance_test_table_16 [id=c7a03df129af49a4a7f1052b1e4a915c]) successfully deleted
I20240503 13:59:23.874853 8948 raft_consensus.cc:1261] T 04c63a06c2e243b9a931b951572da020 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:23.874918 9081 raft_consensus.cc:1261] T 04c63a06c2e243b9a931b951572da020 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:23.875259 9762 consensus_queue.cc:1035] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:23.875404 9791 consensus_queue.cc:1035] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:23.878079 6383 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet 04c63a06c2e243b9a931b951572da020 (table rebalancer_dynamic_table [id=3263b563e6764dcd9d00e2efbd89fc48]). Expected version 1 got 0
I20240503 13:59:23.878418 9879 tablet.cc:1722] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:23.878788 9879 tablet.cc:1620] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:23.878898 6380 catalog_manager.cc:5576] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481) does not have the latest schema for tablet 04c63a06c2e243b9a931b951572da020 (table rebalancer_dynamic_table [id=3263b563e6764dcd9d00e2efbd89fc48]). Expected version 1 got 0
I20240503 13:59:23.880498 9836 tablet.cc:1722] T 04c63a06c2e243b9a931b951572da020 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:23.880792 9836 tablet.cc:1620] T 04c63a06c2e243b9a931b951572da020 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:23.881678 6380 catalog_manager.cc:5576] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481) does not have the latest schema for tablet 04c63a06c2e243b9a931b951572da020 (table rebalancer_dynamic_table [id=3263b563e6764dcd9d00e2efbd89fc48]). Expected version 1 got 0
I20240503 13:59:23.883270 9880 tablet.cc:1722] T 04c63a06c2e243b9a931b951572da020 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:23.883458 6383 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=3263b563e6764dcd9d00e2efbd89fc48] alter complete (version 1)
I20240503 13:59:23.883620 9880 tablet.cc:1620] T 04c63a06c2e243b9a931b951572da020 P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:23.985405 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:23.988305 6791 raft_consensus.cc:3037] T 9253d1943fd74523bafe01c1acf79188 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:23.989949 9725 consensus_queue.cc:1035] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:23.990761 9879 tablet.cc:1722] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:23.991070 9879 tablet.cc:1620] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:23.991958 9880 tablet.cc:1722] T 04c63a06c2e243b9a931b951572da020 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:23.992254 9880 tablet.cc:1620] T 04c63a06c2e243b9a931b951572da020 P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:23.994076 9836 tablet.cc:1722] T 04c63a06c2e243b9a931b951572da020 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:23.994365 9836 tablet.cc:1620] T 04c63a06c2e243b9a931b951572da020 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:23.996371 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=3263b563e6764dcd9d00e2efbd89fc48] alter complete (version 2)
I20240503 13:59:23.999539 9710 consensus_queue.cc:1035] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:24.098454 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:24.103556 9879 tablet.cc:1722] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:24.104420 9836 tablet.cc:1722] T 04c63a06c2e243b9a931b951572da020 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:24.105078 9880 tablet.cc:1722] T 04c63a06c2e243b9a931b951572da020 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:24.109616 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=3263b563e6764dcd9d00e2efbd89fc48] alter complete (version 3)
I20240503 13:59:24.126370 9796 consensus_queue.cc:1035] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:24.129297 9717 consensus_queue.cc:1035] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:24.200345 6380 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:24.205695 9191 tablet_service.cc:1467] Processing CreateTablet for tablet a7e9c80dbf8c41469e54535c20648df4 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=3fa9362ac0d94323a9cd7a56209c3e07]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:24.205775 8927 tablet_service.cc:1467] Processing CreateTablet for tablet a7e9c80dbf8c41469e54535c20648df4 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=3fa9362ac0d94323a9cd7a56209c3e07]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:24.206032 9191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7e9c80dbf8c41469e54535c20648df4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:24.206089 8927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7e9c80dbf8c41469e54535c20648df4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:24.206511 9061 tablet_service.cc:1467] Processing CreateTablet for tablet 96d1a236cdb7406090021842d8dbc5a0 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=3fa9362ac0d94323a9cd7a56209c3e07]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:24.206558 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 0b397d0460c94438a773be184312b60f (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=3fa9362ac0d94323a9cd7a56209c3e07]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:24.206748 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96d1a236cdb7406090021842d8dbc5a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:24.206933 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b397d0460c94438a773be184312b60f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:24.208320 9857 tablet_bootstrap.cc:492] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:24.208765 6507 tablet_service.cc:1467] Processing CreateTablet for tablet 96d1a236cdb7406090021842d8dbc5a0 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=3fa9362ac0d94323a9cd7a56209c3e07]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:24.209021 6507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96d1a236cdb7406090021842d8dbc5a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:24.209138 6508 tablet_service.cc:1467] Processing CreateTablet for tablet a7e9c80dbf8c41469e54535c20648df4 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=3fa9362ac0d94323a9cd7a56209c3e07]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:24.209301 9857 tablet_bootstrap.cc:654] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:24.209332 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7e9c80dbf8c41469e54535c20648df4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:24.209890 9859 tablet_bootstrap.cc:492] T a7e9c80dbf8c41469e54535c20648df4 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:24.210742 9859 tablet_bootstrap.cc:654] T a7e9c80dbf8c41469e54535c20648df4 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:24.211062 9849 tablet_bootstrap.cc:492] T 96d1a236cdb7406090021842d8dbc5a0 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:24.211736 9323 tablet_service.cc:1467] Processing CreateTablet for tablet 0b397d0460c94438a773be184312b60f (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=3fa9362ac0d94323a9cd7a56209c3e07]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:24.211967 9849 tablet_bootstrap.cc:654] T 96d1a236cdb7406090021842d8dbc5a0 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:24.212033 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b397d0460c94438a773be184312b60f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:24.212137 9857 tablet_bootstrap.cc:492] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:24.212205 9857 ts_tablet_manager.cc:1367] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:24.212658 9857 raft_consensus.cc:348] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:24.212754 9857 raft_consensus.cc:374] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:24.212715 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 0b397d0460c94438a773be184312b60f (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=3fa9362ac0d94323a9cd7a56209c3e07]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:24.212783 9857 raft_consensus.cc:729] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:24.212900 9857 consensus_queue.cc:260] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:24.212958 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b397d0460c94438a773be184312b60f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:24.213085 8927 tablet_service.cc:1467] Processing CreateTablet for tablet 96d1a236cdb7406090021842d8dbc5a0 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=3fa9362ac0d94323a9cd7a56209c3e07]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:24.213165 9857 ts_tablet_manager.cc:1398] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:24.213268 8927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96d1a236cdb7406090021842d8dbc5a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:24.214004 9889 tablet_bootstrap.cc:492] T 96d1a236cdb7406090021842d8dbc5a0 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:24.214788 9889 tablet_bootstrap.cc:654] T 96d1a236cdb7406090021842d8dbc5a0 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:24.215090 9892 tablet_bootstrap.cc:492] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:24.215721 9849 tablet_bootstrap.cc:492] T 96d1a236cdb7406090021842d8dbc5a0 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:24.215755 9889 tablet_bootstrap.cc:492] T 96d1a236cdb7406090021842d8dbc5a0 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:24.215806 9849 ts_tablet_manager.cc:1367] T 96d1a236cdb7406090021842d8dbc5a0 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:24.215829 9889 ts_tablet_manager.cc:1367] T 96d1a236cdb7406090021842d8dbc5a0 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:24.216024 9892 tablet_bootstrap.cc:654] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:24.216257 9889 raft_consensus.cc:348] T 96d1a236cdb7406090021842d8dbc5a0 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:24.216363 9889 raft_consensus.cc:374] T 96d1a236cdb7406090021842d8dbc5a0 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:24.216300 9849 raft_consensus.cc:348] T 96d1a236cdb7406090021842d8dbc5a0 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:24.216398 9889 raft_consensus.cc:729] T 96d1a236cdb7406090021842d8dbc5a0 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:24.216413 9849 raft_consensus.cc:374] T 96d1a236cdb7406090021842d8dbc5a0 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:24.216444 9849 raft_consensus.cc:729] T 96d1a236cdb7406090021842d8dbc5a0 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:24.216387 6383 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:24.216502 6383 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:24.216508 9889 consensus_queue.cc:260] T 96d1a236cdb7406090021842d8dbc5a0 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:24.216555 9849 consensus_queue.cc:260] T 96d1a236cdb7406090021842d8dbc5a0 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:24.216773 9889 ts_tablet_manager.cc:1398] T 96d1a236cdb7406090021842d8dbc5a0 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:24.216853 9849 ts_tablet_manager.cc:1398] T 96d1a236cdb7406090021842d8dbc5a0 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:24.216935 9889 tablet_bootstrap.cc:492] T a7e9c80dbf8c41469e54535c20648df4 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:24.217021 9849 tablet_bootstrap.cc:492] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:24.217368 9859 tablet_bootstrap.cc:492] T a7e9c80dbf8c41469e54535c20648df4 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:24.217442 9859 ts_tablet_manager.cc:1367] T a7e9c80dbf8c41469e54535c20648df4 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:59:24.217780 9889 tablet_bootstrap.cc:654] T a7e9c80dbf8c41469e54535c20648df4 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:24.217897 9859 raft_consensus.cc:348] T a7e9c80dbf8c41469e54535c20648df4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:24.217998 9859 raft_consensus.cc:374] T a7e9c80dbf8c41469e54535c20648df4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:24.218027 9859 raft_consensus.cc:729] T a7e9c80dbf8c41469e54535c20648df4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:24.218145 9859 consensus_queue.cc:260] T a7e9c80dbf8c41469e54535c20648df4 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:24.218240 9849 tablet_bootstrap.cc:654] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:24.218407 9859 ts_tablet_manager.cc:1398] T a7e9c80dbf8c41469e54535c20648df4 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:24.218566 9859 tablet_bootstrap.cc:492] T 0b397d0460c94438a773be184312b60f P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:24.219439 9859 tablet_bootstrap.cc:654] T 0b397d0460c94438a773be184312b60f P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:24.219514 9849 tablet_bootstrap.cc:492] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:24.219602 9849 ts_tablet_manager.cc:1367] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 13:59:24.219740 9892 tablet_bootstrap.cc:492] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:24.219813 9892 ts_tablet_manager.cc:1367] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:24.220050 9849 raft_consensus.cc:348] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:24.220136 9849 raft_consensus.cc:374] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:24.220166 9849 raft_consensus.cc:729] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:24.220240 9892 raft_consensus.cc:348] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:24.220286 9849 consensus_queue.cc:260] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:24.220331 9892 raft_consensus.cc:374] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:24.220363 9892 raft_consensus.cc:729] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:24.220484 9892 consensus_queue.cc:260] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:24.220542 9849 ts_tablet_manager.cc:1398] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:24.220731 9892 ts_tablet_manager.cc:1398] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:24.222311 6383 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 04c63a06c2e243b9a931b951572da020
I20240503 13:59:24.223268 9191 tablet_service.cc:1514] Processing DeleteTablet for tablet 04c63a06c2e243b9a931b951572da020 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:24 UTC) from {username='slave'} at 127.0.0.1:57960
I20240503 13:59:24.223690 9061 tablet_service.cc:1514] Processing DeleteTablet for tablet 04c63a06c2e243b9a931b951572da020 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:24 UTC) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:24.224045 9889 tablet_bootstrap.cc:492] T a7e9c80dbf8c41469e54535c20648df4 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:24.224130 9889 ts_tablet_manager.cc:1367] T a7e9c80dbf8c41469e54535c20648df4 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:59:24.224874 9889 raft_consensus.cc:348] T a7e9c80dbf8c41469e54535c20648df4 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:24.225024 9889 raft_consensus.cc:374] T a7e9c80dbf8c41469e54535c20648df4 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:24.225062 9889 raft_consensus.cc:729] T a7e9c80dbf8c41469e54535c20648df4 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:24.225175 9889 consensus_queue.cc:260] T a7e9c80dbf8c41469e54535c20648df4 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:24.225490 9889 ts_tablet_manager.cc:1398] T a7e9c80dbf8c41469e54535c20648df4 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:24.226126 9859 tablet_bootstrap.cc:492] T 0b397d0460c94438a773be184312b60f P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:24.226212 9859 ts_tablet_manager.cc:1367] T 0b397d0460c94438a773be184312b60f P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:59:24.226666 9859 raft_consensus.cc:348] T 0b397d0460c94438a773be184312b60f P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:24.226805 9859 raft_consensus.cc:374] T 0b397d0460c94438a773be184312b60f P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:24.226845 9859 raft_consensus.cc:729] T 0b397d0460c94438a773be184312b60f P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:24.226975 9859 consensus_queue.cc:260] T 0b397d0460c94438a773be184312b60f P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:24.227236 9859 ts_tablet_manager.cc:1398] T 0b397d0460c94438a773be184312b60f P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:24.227355 8928 tablet_service.cc:1514] Processing DeleteTablet for tablet 04c63a06c2e243b9a931b951572da020 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:24 UTC) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:24.227413 9859 tablet_bootstrap.cc:492] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:24.228303 9859 tablet_bootstrap.cc:654] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:24.230104 9899 tablet_replica.cc:330] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 13:59:24.230273 9899 raft_consensus.cc:2229] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:24.230593 9899 raft_consensus.cc:2258] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:24.231231 9899 ts_tablet_manager.cc:1875] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:24.231339 9900 tablet_replica.cc:330] T 04c63a06c2e243b9a931b951572da020 P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:24.231487 9900 raft_consensus.cc:2229] T 04c63a06c2e243b9a931b951572da020 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:24.231631 9900 raft_consensus.cc:2258] T 04c63a06c2e243b9a931b951572da020 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:24.231873 9902 tablet_replica.cc:330] T 04c63a06c2e243b9a931b951572da020 P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:24.232000 9902 raft_consensus.cc:2229] T 04c63a06c2e243b9a931b951572da020 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:24.232115 9902 raft_consensus.cc:2258] T 04c63a06c2e243b9a931b951572da020 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:24.232933 9859 tablet_bootstrap.cc:492] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:24.233008 9900 ts_tablet_manager.cc:1875] T 04c63a06c2e243b9a931b951572da020 P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:24.233024 9859 ts_tablet_manager.cc:1367] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 13:59:24.233501 9859 raft_consensus.cc:348] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:24.233628 9859 raft_consensus.cc:374] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:24.233670 9859 raft_consensus.cc:729] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:24.233794 9859 consensus_queue.cc:260] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:24.234086 9859 ts_tablet_manager.cc:1398] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:24.233484 9902 ts_tablet_manager.cc:1875] T 04c63a06c2e243b9a931b951572da020 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:24.235571 9899 ts_tablet_manager.cc:1888] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:24.236299 9899 log.cc:1196] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/04c63a06c2e243b9a931b951572da020
I20240503 13:59:24.236555 9902 ts_tablet_manager.cc:1888] T 04c63a06c2e243b9a931b951572da020 P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:24.236630 9899 ts_tablet_manager.cc:1909] T 04c63a06c2e243b9a931b951572da020 P ab1d21935d6049888eb0ceccc3c9d735: Deleting consensus metadata
I20240503 13:59:24.236668 9902 log.cc:1196] T 04c63a06c2e243b9a931b951572da020 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/04c63a06c2e243b9a931b951572da020
I20240503 13:59:24.237000 9902 ts_tablet_manager.cc:1909] T 04c63a06c2e243b9a931b951572da020 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting consensus metadata
I20240503 13:59:24.237478 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet 04c63a06c2e243b9a931b951572da020 (table rebalancer_dynamic_table_new_name [id=3263b563e6764dcd9d00e2efbd89fc48]) successfully deleted
I20240503 13:59:24.237623 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 04c63a06c2e243b9a931b951572da020 (table rebalancer_dynamic_table_new_name [id=3263b563e6764dcd9d00e2efbd89fc48]) successfully deleted
I20240503 13:59:24.238904 9900 ts_tablet_manager.cc:1888] T 04c63a06c2e243b9a931b951572da020 P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:24.238982 9900 log.cc:1196] T 04c63a06c2e243b9a931b951572da020 P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/04c63a06c2e243b9a931b951572da020
I20240503 13:59:24.239226 9900 ts_tablet_manager.cc:1909] T 04c63a06c2e243b9a931b951572da020 P 8d574fb4a89d4eea947c859470d44dd4: Deleting consensus metadata
I20240503 13:59:24.239923 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet 04c63a06c2e243b9a931b951572da020 (table rebalancer_dynamic_table_new_name [id=3263b563e6764dcd9d00e2efbd89fc48]) successfully deleted
I20240503 13:59:24.253803 9762 raft_consensus.cc:482] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:24.253897 9762 raft_consensus.cc:504] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:24.254244 9762 leader_election.cc:288] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:24.254585 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7e9c80dbf8c41469e54535c20648df4" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:24.254648 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7e9c80dbf8c41469e54535c20648df4" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:24.254706 8948 raft_consensus.cc:2454] T a7e9c80dbf8c41469e54535c20648df4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:24.254793 6528 raft_consensus.cc:2454] T a7e9c80dbf8c41469e54535c20648df4 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:24.255329 9147 leader_election.cc:302] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:24.255481 9762 raft_consensus.cc:2781] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:24.255535 9762 raft_consensus.cc:482] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:24.255576 9762 raft_consensus.cc:3037] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:24.256520 9762 raft_consensus.cc:504] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:24.256815 9762 leader_election.cc:288] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Requested vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:24.257087 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7e9c80dbf8c41469e54535c20648df4" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:24.257171 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7e9c80dbf8c41469e54535c20648df4" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:24.257217 8948 raft_consensus.cc:3037] T a7e9c80dbf8c41469e54535c20648df4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:24.257287 6528 raft_consensus.cc:3037] T a7e9c80dbf8c41469e54535c20648df4 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:24.257993 6528 raft_consensus.cc:2454] T a7e9c80dbf8c41469e54535c20648df4 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:24.258005 8948 raft_consensus.cc:2454] T a7e9c80dbf8c41469e54535c20648df4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:24.258241 9147 leader_election.cc:302] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:24.258436 9762 raft_consensus.cc:2781] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:24.258536 9762 raft_consensus.cc:686] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 13:59:24.258661 9762 consensus_queue.cc:237] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:24.260258 6383 catalog_manager.cc:5484] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:24.276767 9838 raft_consensus.cc:482] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:24.276880 9838 raft_consensus.cc:504] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:24.277308 9838 leader_election.cc:288] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:24.277477 9762 consensus_queue.cc:1035] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:24.277694 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b397d0460c94438a773be184312b60f" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:24.277837 9081 raft_consensus.cc:2454] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:24.278069 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b397d0460c94438a773be184312b60f" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:24.278096 9278 leader_election.cc:302] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:24.278205 8948 raft_consensus.cc:2454] T 0b397d0460c94438a773be184312b60f P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:24.278264 9838 raft_consensus.cc:2781] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:24.278339 9838 raft_consensus.cc:482] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:24.278388 9838 raft_consensus.cc:3037] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:24.279582 9838 raft_consensus.cc:504] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:24.279947 9838 leader_election.cc:288] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:24.280455 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b397d0460c94438a773be184312b60f" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:24.280577 9081 raft_consensus.cc:3037] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:24.281255 9081 raft_consensus.cc:2454] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:24.281618 9278 leader_election.cc:302] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:24.281750 9838 raft_consensus.cc:2781] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:24.281841 9838 raft_consensus.cc:686] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:24.281970 9838 consensus_queue.cc:237] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:24.282264 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b397d0460c94438a773be184312b60f" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:24.282369 8948 raft_consensus.cc:3037] T 0b397d0460c94438a773be184312b60f P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:24.283309 8948 raft_consensus.cc:2454] T 0b397d0460c94438a773be184312b60f P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:24.284126 6383 catalog_manager.cc:5484] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:24.285727 9762 consensus_queue.cc:1035] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:24.324162 6386 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:24.329072 9191 tablet_service.cc:1467] Processing CreateTablet for tablet df050fcf850345e38595f3312a22f60a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=db91e8e19d8344f198ca8a7d418b5be2]), partition=
I20240503 13:59:24.329138 9323 tablet_service.cc:1467] Processing CreateTablet for tablet df050fcf850345e38595f3312a22f60a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=db91e8e19d8344f198ca8a7d418b5be2]), partition=
I20240503 13:59:24.329191 6771 tablet_service.cc:1467] Processing CreateTablet for tablet df050fcf850345e38595f3312a22f60a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=db91e8e19d8344f198ca8a7d418b5be2]), partition=
I20240503 13:59:24.329381 9191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df050fcf850345e38595f3312a22f60a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:24.329416 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df050fcf850345e38595f3312a22f60a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:24.329407 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df050fcf850345e38595f3312a22f60a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:24.331594 9857 tablet_bootstrap.cc:492] T df050fcf850345e38595f3312a22f60a P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:24.331665 9892 tablet_bootstrap.cc:492] T df050fcf850345e38595f3312a22f60a P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:24.331732 9909 tablet_bootstrap.cc:492] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:24.332530 9892 tablet_bootstrap.cc:654] T df050fcf850345e38595f3312a22f60a P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:24.332548 9909 tablet_bootstrap.cc:654] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:24.332593 9857 tablet_bootstrap.cc:654] T df050fcf850345e38595f3312a22f60a P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:24.333729 9857 tablet_bootstrap.cc:492] T df050fcf850345e38595f3312a22f60a P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:24.333787 9892 tablet_bootstrap.cc:492] T df050fcf850345e38595f3312a22f60a P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:24.333806 9857 ts_tablet_manager.cc:1367] T df050fcf850345e38595f3312a22f60a P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:24.333860 9892 ts_tablet_manager.cc:1367] T df050fcf850345e38595f3312a22f60a P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:24.334203 9857 raft_consensus.cc:348] T df050fcf850345e38595f3312a22f60a P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:24.334288 9857 raft_consensus.cc:374] T df050fcf850345e38595f3312a22f60a P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:24.334309 9857 raft_consensus.cc:729] T df050fcf850345e38595f3312a22f60a P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:24.334302 9892 raft_consensus.cc:348] T df050fcf850345e38595f3312a22f60a P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:24.334404 9892 raft_consensus.cc:374] T df050fcf850345e38595f3312a22f60a P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:24.334412 9857 consensus_queue.cc:260] T df050fcf850345e38595f3312a22f60a P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:24.334451 9892 raft_consensus.cc:729] T df050fcf850345e38595f3312a22f60a P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:24.334591 9892 consensus_queue.cc:260] T df050fcf850345e38595f3312a22f60a P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:24.334638 9857 ts_tablet_manager.cc:1398] T df050fcf850345e38595f3312a22f60a P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:24.334892 9892 ts_tablet_manager.cc:1398] T df050fcf850345e38595f3312a22f60a P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:24.336637 9909 tablet_bootstrap.cc:492] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:24.336737 9909 ts_tablet_manager.cc:1367] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:59:24.337123 9909 raft_consensus.cc:348] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:24.337210 9909 raft_consensus.cc:374] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:24.337231 9909 raft_consensus.cc:729] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:24.337332 9909 consensus_queue.cc:260] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:24.337651 9909 ts_tablet_manager.cc:1398] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:24.361567 6386 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240503 13:59:24.361724 6386 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240503 13:59:24.364104 6386 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet b05fc3d7abb04aa086ec4caf7eabb6ab
I20240503 13:59:24.364462 6386 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 0162cfee86ed4f3394e6829a405037fe
I20240503 13:59:24.364579 6771 tablet_service.cc:1514] Processing DeleteTablet for tablet b05fc3d7abb04aa086ec4caf7eabb6ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:24 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:24.364750 9731 tablet_replica.cc:330] T b05fc3d7abb04aa086ec4caf7eabb6ab P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:24.364758 6640 tablet_service.cc:1514] Processing DeleteTablet for tablet b05fc3d7abb04aa086ec4caf7eabb6ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:24 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:24.364850 6771 tablet_service.cc:1514] Processing DeleteTablet for tablet 0162cfee86ed4f3394e6829a405037fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:24 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:24.365008 6386 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 0d75271c2d2d4aa88f7e499898300782
I20240503 13:59:24.365059 9540 tablet_replica.cc:330] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:24.365172 9540 raft_consensus.cc:2229] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:24.365151 6508 tablet_service.cc:1514] Processing DeleteTablet for tablet b05fc3d7abb04aa086ec4caf7eabb6ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:24 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:24.364902 9731 raft_consensus.cc:2229] T b05fc3d7abb04aa086ec4caf7eabb6ab P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:24.365319 9541 tablet_replica.cc:330] T b05fc3d7abb04aa086ec4caf7eabb6ab P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:24.365346 9731 raft_consensus.cc:2258] T b05fc3d7abb04aa086ec4caf7eabb6ab P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:24.365376 9540 raft_consensus.cc:2258] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:24.365437 9541 raft_consensus.cc:2229] T b05fc3d7abb04aa086ec4caf7eabb6ab P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:24.365581 9541 raft_consensus.cc:2258] T b05fc3d7abb04aa086ec4caf7eabb6ab P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:24.365715 6507 tablet_service.cc:1514] Processing DeleteTablet for tablet 0162cfee86ed4f3394e6829a405037fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:24 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:24.365892 9731 ts_tablet_manager.cc:1875] T b05fc3d7abb04aa086ec4caf7eabb6ab P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:24.365919 9540 ts_tablet_manager.cc:1875] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:24.366055 9541 ts_tablet_manager.cc:1875] T b05fc3d7abb04aa086ec4caf7eabb6ab P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:24.366207 6771 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d75271c2d2d4aa88f7e499898300782 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:24 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:24.366430 6901 tablet_service.cc:1514] Processing DeleteTablet for tablet 0162cfee86ed4f3394e6829a405037fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:24 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:24.366959 6640 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d75271c2d2d4aa88f7e499898300782 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:24 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:24.367208 6507 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d75271c2d2d4aa88f7e499898300782 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:24 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:24.367403 9875 tablet_replica.cc:330] T 0162cfee86ed4f3394e6829a405037fe P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:24.367563 9875 raft_consensus.cc:2229] T 0162cfee86ed4f3394e6829a405037fe P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:24.367715 9875 raft_consensus.cc:2258] T 0162cfee86ed4f3394e6829a405037fe P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:24.368286 9875 ts_tablet_manager.cc:1875] T 0162cfee86ed4f3394e6829a405037fe P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:24.368391 9731 ts_tablet_manager.cc:1888] T b05fc3d7abb04aa086ec4caf7eabb6ab P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:24.368409 9540 ts_tablet_manager.cc:1888] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:24.368449 9731 log.cc:1196] T b05fc3d7abb04aa086ec4caf7eabb6ab P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/b05fc3d7abb04aa086ec4caf7eabb6ab
I20240503 13:59:24.368471 9540 log.cc:1196] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/b05fc3d7abb04aa086ec4caf7eabb6ab
I20240503 13:59:24.368716 9731 ts_tablet_manager.cc:1909] T b05fc3d7abb04aa086ec4caf7eabb6ab P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:24.368744 9540 ts_tablet_manager.cc:1909] T b05fc3d7abb04aa086ec4caf7eabb6ab P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:24.369042 9541 ts_tablet_manager.cc:1888] T b05fc3d7abb04aa086ec4caf7eabb6ab P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:24.369107 9541 log.cc:1196] T b05fc3d7abb04aa086ec4caf7eabb6ab P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/b05fc3d7abb04aa086ec4caf7eabb6ab
I20240503 13:59:24.369349 9541 ts_tablet_manager.cc:1909] T b05fc3d7abb04aa086ec4caf7eabb6ab P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:24.369501 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet b05fc3d7abb04aa086ec4caf7eabb6ab (table rebalance_test_table_18 [id=3057a1d5b4be45e49902b304f0a52e14]) successfully deleted
I20240503 13:59:24.369530 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet b05fc3d7abb04aa086ec4caf7eabb6ab (table rebalance_test_table_18 [id=3057a1d5b4be45e49902b304f0a52e14]) successfully deleted
I20240503 13:59:24.369715 9731 tablet_replica.cc:330] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:24.369827 9731 raft_consensus.cc:2229] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:24.369875 9540 tablet_replica.cc:330] T 0d75271c2d2d4aa88f7e499898300782 P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:24.370021 9731 raft_consensus.cc:2258] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:24.370023 9540 raft_consensus.cc:2229] T 0d75271c2d2d4aa88f7e499898300782 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:24.370023 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet b05fc3d7abb04aa086ec4caf7eabb6ab (table rebalance_test_table_18 [id=3057a1d5b4be45e49902b304f0a52e14]) successfully deleted
I20240503 13:59:24.370149 9540 raft_consensus.cc:2258] T 0d75271c2d2d4aa88f7e499898300782 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:24.370517 9731 ts_tablet_manager.cc:1875] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:24.370522 9875 ts_tablet_manager.cc:1888] T 0162cfee86ed4f3394e6829a405037fe P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:24.370584 9875 log.cc:1196] T 0162cfee86ed4f3394e6829a405037fe P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/0162cfee86ed4f3394e6829a405037fe
I20240503 13:59:24.370602 9540 ts_tablet_manager.cc:1875] T 0d75271c2d2d4aa88f7e499898300782 P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:24.370771 9541 tablet_replica.cc:330] T 0162cfee86ed4f3394e6829a405037fe P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:24.370826 9875 ts_tablet_manager.cc:1909] T 0162cfee86ed4f3394e6829a405037fe P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:24.370896 9541 raft_consensus.cc:2229] T 0162cfee86ed4f3394e6829a405037fe P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:24.371008 9541 raft_consensus.cc:2258] T 0162cfee86ed4f3394e6829a405037fe P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:24.371464 9541 ts_tablet_manager.cc:1875] T 0162cfee86ed4f3394e6829a405037fe P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:24.372370 9731 ts_tablet_manager.cc:1888] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:24.372435 9731 log.cc:1196] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/0162cfee86ed4f3394e6829a405037fe
I20240503 13:59:24.372434 9540 ts_tablet_manager.cc:1888] T 0d75271c2d2d4aa88f7e499898300782 P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:24.372503 9540 log.cc:1196] T 0d75271c2d2d4aa88f7e499898300782 P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/0d75271c2d2d4aa88f7e499898300782
I20240503 13:59:24.372704 9731 ts_tablet_manager.cc:1909] T 0162cfee86ed4f3394e6829a405037fe P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:24.372774 9540 ts_tablet_manager.cc:1909] T 0d75271c2d2d4aa88f7e499898300782 P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:24.373361 9731 tablet_replica.cc:330] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:24.373459 9541 ts_tablet_manager.cc:1888] T 0162cfee86ed4f3394e6829a405037fe P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:24.373484 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet 0d75271c2d2d4aa88f7e499898300782 (table rebalance_test_table_18 [id=3057a1d5b4be45e49902b304f0a52e14]) successfully deleted
I20240503 13:59:24.373540 9541 log.cc:1196] T 0162cfee86ed4f3394e6829a405037fe P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0162cfee86ed4f3394e6829a405037fe
I20240503 13:59:24.373538 9731 raft_consensus.cc:2229] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:24.373490 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet 0162cfee86ed4f3394e6829a405037fe (table rebalance_test_table_18 [id=3057a1d5b4be45e49902b304f0a52e14]) successfully deleted
I20240503 13:59:24.373720 9731 raft_consensus.cc:2258] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:24.373806 9541 ts_tablet_manager.cc:1909] T 0162cfee86ed4f3394e6829a405037fe P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:24.374128 9731 ts_tablet_manager.cc:1875] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:24.374255 9541 tablet_replica.cc:330] T 0d75271c2d2d4aa88f7e499898300782 P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:24.374361 9541 raft_consensus.cc:2229] T 0d75271c2d2d4aa88f7e499898300782 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:24.374459 9541 raft_consensus.cc:2258] T 0d75271c2d2d4aa88f7e499898300782 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:24.374886 9541 ts_tablet_manager.cc:1875] T 0d75271c2d2d4aa88f7e499898300782 P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:24.375674 9731 ts_tablet_manager.cc:1888] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:24.375736 9731 log.cc:1196] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/0d75271c2d2d4aa88f7e499898300782
I20240503 13:59:24.376015 9731 ts_tablet_manager.cc:1909] T 0d75271c2d2d4aa88f7e499898300782 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:24.376368 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 0162cfee86ed4f3394e6829a405037fe (table rebalance_test_table_18 [id=3057a1d5b4be45e49902b304f0a52e14]) successfully deleted
I20240503 13:59:24.376513 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet 0162cfee86ed4f3394e6829a405037fe (table rebalance_test_table_18 [id=3057a1d5b4be45e49902b304f0a52e14]) successfully deleted
I20240503 13:59:24.376586 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet 0d75271c2d2d4aa88f7e499898300782 (table rebalance_test_table_18 [id=3057a1d5b4be45e49902b304f0a52e14]) successfully deleted
I20240503 13:59:24.376783 9541 ts_tablet_manager.cc:1888] T 0d75271c2d2d4aa88f7e499898300782 P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:59:24.376852 9541 log.cc:1196] T 0d75271c2d2d4aa88f7e499898300782 P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0d75271c2d2d4aa88f7e499898300782
I20240503 13:59:24.377035 9541 ts_tablet_manager.cc:1909] T 0d75271c2d2d4aa88f7e499898300782 P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:24.377516 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 0d75271c2d2d4aa88f7e499898300782 (table rebalance_test_table_18 [id=3057a1d5b4be45e49902b304f0a52e14]) successfully deleted
I20240503 13:59:24.430024 9717 raft_consensus.cc:482] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:24.430145 9717 raft_consensus.cc:504] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:24.430589 9717 leader_election.cc:288] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:24.430940 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96d1a236cdb7406090021842d8dbc5a0" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:24.431089 9080 raft_consensus.cc:2454] T 96d1a236cdb7406090021842d8dbc5a0 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:24.431387 8883 leader_election.cc:302] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:24.431540 9717 raft_consensus.cc:2781] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:24.431607 9717 raft_consensus.cc:482] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:24.431663 9717 raft_consensus.cc:3037] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:24.432567 9717 raft_consensus.cc:504] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:24.432871 9717 leader_election.cc:288] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:24.433127 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96d1a236cdb7406090021842d8dbc5a0" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:24.433230 9080 raft_consensus.cc:3037] T 96d1a236cdb7406090021842d8dbc5a0 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:24.434116 9080 raft_consensus.cc:2454] T 96d1a236cdb7406090021842d8dbc5a0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:24.434370 8883 leader_election.cc:302] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:24.434527 9717 raft_consensus.cc:2781] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:24.434603 9717 raft_consensus.cc:686] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 13:59:24.434609 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96d1a236cdb7406090021842d8dbc5a0" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:24.434635 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96d1a236cdb7406090021842d8dbc5a0" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:24.434724 6528 raft_consensus.cc:2454] T 96d1a236cdb7406090021842d8dbc5a0 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:24.434728 9717 consensus_queue.cc:237] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:24.436266 6386 catalog_manager.cc:5484] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:24.541653 9737 raft_consensus.cc:482] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:24.541776 9737 raft_consensus.cc:504] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:24.542275 9737 leader_election.cc:288] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:24.546757 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df050fcf850345e38595f3312a22f60a" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:24.546757 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df050fcf850345e38595f3312a22f60a" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:24.546896 9205 raft_consensus.cc:2454] T df050fcf850345e38595f3312a22f60a P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:24.546929 9343 raft_consensus.cc:2454] T df050fcf850345e38595f3312a22f60a P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:24.547135 6726 leader_election.cc:302] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:24.547329 9737 raft_consensus.cc:2781] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:24.547390 9737 raft_consensus.cc:482] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:24.547417 9737 raft_consensus.cc:3037] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:24.548343 9737 raft_consensus.cc:504] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:24.548683 9737 leader_election.cc:288] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:24.548990 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df050fcf850345e38595f3312a22f60a" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:24.548983 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df050fcf850345e38595f3312a22f60a" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:24.549095 9343 raft_consensus.cc:3037] T df050fcf850345e38595f3312a22f60a P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:24.549101 9205 raft_consensus.cc:3037] T df050fcf850345e38595f3312a22f60a P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:24.549840 9343 raft_consensus.cc:2454] T df050fcf850345e38595f3312a22f60a P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:24.549851 9205 raft_consensus.cc:2454] T df050fcf850345e38595f3312a22f60a P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:24.550069 6727 leader_election.cc:302] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:24.550238 9737 raft_consensus.cc:2781] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:24.550407 9737 raft_consensus.cc:686] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:24.550527 9737 consensus_queue.cc:237] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:24.552125 6386 catalog_manager.cc:5484] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:24.663631 6386 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:24.667964 9342 raft_consensus.cc:1261] T df050fcf850345e38595f3312a22f60a P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:24.668392 9920 consensus_queue.cc:1035] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:24.668571 9205 raft_consensus.cc:1261] T df050fcf850345e38595f3312a22f60a P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:24.669023 9920 consensus_queue.cc:1035] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:24.671192 6383 catalog_manager.cc:5576] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091) does not have the latest schema for tablet df050fcf850345e38595f3312a22f60a (table rebalancer_dynamic_table [id=db91e8e19d8344f198ca8a7d418b5be2]). Expected version 1 got 0
I20240503 13:59:24.671710 6386 catalog_manager.cc:5576] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931) does not have the latest schema for tablet df050fcf850345e38595f3312a22f60a (table rebalancer_dynamic_table [id=db91e8e19d8344f198ca8a7d418b5be2]). Expected version 1 got 0
I20240503 13:59:24.672245 9930 tablet.cc:1722] T df050fcf850345e38595f3312a22f60a P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:24.672794 9930 tablet.cc:1620] T df050fcf850345e38595f3312a22f60a P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:24.673280 6386 catalog_manager.cc:5576] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365) does not have the latest schema for tablet df050fcf850345e38595f3312a22f60a (table rebalancer_dynamic_table [id=db91e8e19d8344f198ca8a7d418b5be2]). Expected version 1 got 0
I20240503 13:59:24.673372 9931 tablet.cc:1722] T df050fcf850345e38595f3312a22f60a P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:24.673671 9931 tablet.cc:1620] T df050fcf850345e38595f3312a22f60a P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:24.673964 9929 tablet.cc:1722] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:24.675849 9929 tablet.cc:1620] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64: MemRowSet was empty: no flush needed.
I20240503 13:59:24.676613 6383 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=db91e8e19d8344f198ca8a7d418b5be2] alter complete (version 1)
I20240503 13:59:24.769924 9762 consensus_queue.cc:1035] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:24.772919 9828 consensus_queue.cc:1035] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:24.779358 6385 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:24.784364 9929 tablet.cc:1722] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:24.784685 9929 tablet.cc:1620] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64: MemRowSet was empty: no flush needed.
I20240503 13:59:24.784910 9930 tablet.cc:1722] T df050fcf850345e38595f3312a22f60a P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:24.785269 9930 tablet.cc:1620] T df050fcf850345e38595f3312a22f60a P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:24.785475 9931 tablet.cc:1722] T df050fcf850345e38595f3312a22f60a P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:24.788758 9931 tablet.cc:1620] T df050fcf850345e38595f3312a22f60a P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:24.789073 6385 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=db91e8e19d8344f198ca8a7d418b5be2] alter complete (version 2)
I20240503 13:59:24.835479 9838 consensus_queue.cc:1035] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:24.839812 9838 consensus_queue.cc:1035] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:24.851884 9923 consensus_queue.cc:1035] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:24.854236 6527 raft_consensus.cc:3037] T 96d1a236cdb7406090021842d8dbc5a0 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:24.855859 9923 consensus_queue.cc:1035] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:24.890561 6385 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:24.895998 9931 tablet.cc:1722] T df050fcf850345e38595f3312a22f60a P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:24.896163 9930 tablet.cc:1722] T df050fcf850345e38595f3312a22f60a P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:24.897068 9929 tablet.cc:1722] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:24.900714 6383 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=db91e8e19d8344f198ca8a7d418b5be2] alter complete (version 3)
I20240503 13:59:24.959424 6385 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:24.964721 9191 tablet_service.cc:1467] Processing CreateTablet for tablet f6053da8b5a64536b738465aa6a3db10 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b4dfb7bd2b7f40e9ad8e6e5c28e9ee31]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:24.965111 9191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6053da8b5a64536b738465aa6a3db10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:24.965255 6507 tablet_service.cc:1467] Processing CreateTablet for tablet f6053da8b5a64536b738465aa6a3db10 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b4dfb7bd2b7f40e9ad8e6e5c28e9ee31]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:24.965355 9190 tablet_service.cc:1467] Processing CreateTablet for tablet 62c77ec99cf443d2b7d48b12c8855883 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b4dfb7bd2b7f40e9ad8e6e5c28e9ee31]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:24.965553 6507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6053da8b5a64536b738465aa6a3db10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:24.965597 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c77ec99cf443d2b7d48b12c8855883. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:24.967154 9942 tablet_bootstrap.cc:492] T f6053da8b5a64536b738465aa6a3db10 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:24.967142 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 62c77ec99cf443d2b7d48b12c8855883 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b4dfb7bd2b7f40e9ad8e6e5c28e9ee31]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:24.967379 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c77ec99cf443d2b7d48b12c8855883. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:24.968094 8928 tablet_service.cc:1467] Processing CreateTablet for tablet f6053da8b5a64536b738465aa6a3db10 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b4dfb7bd2b7f40e9ad8e6e5c28e9ee31]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:24.968185 9323 tablet_service.cc:1467] Processing CreateTablet for tablet 61dfb1d40ee444368470a0dafc3179a4 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b4dfb7bd2b7f40e9ad8e6e5c28e9ee31]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:24.968310 9942 tablet_bootstrap.cc:654] T f6053da8b5a64536b738465aa6a3db10 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:24.968370 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6053da8b5a64536b738465aa6a3db10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:24.968425 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61dfb1d40ee444368470a0dafc3179a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:24.970484 9945 tablet_bootstrap.cc:492] T f6053da8b5a64536b738465aa6a3db10 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:24.970484 9946 tablet_bootstrap.cc:492] T 61dfb1d40ee444368470a0dafc3179a4 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:24.970716 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 61dfb1d40ee444368470a0dafc3179a4 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b4dfb7bd2b7f40e9ad8e6e5c28e9ee31]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:24.970988 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61dfb1d40ee444368470a0dafc3179a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:24.971112 9061 tablet_service.cc:1467] Processing CreateTablet for tablet 62c77ec99cf443d2b7d48b12c8855883 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b4dfb7bd2b7f40e9ad8e6e5c28e9ee31]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:24.971388 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c77ec99cf443d2b7d48b12c8855883. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:24.971626 9946 tablet_bootstrap.cc:654] T 61dfb1d40ee444368470a0dafc3179a4 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:24.971661 9945 tablet_bootstrap.cc:654] T f6053da8b5a64536b738465aa6a3db10 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:24.972899 9946 tablet_bootstrap.cc:492] T 61dfb1d40ee444368470a0dafc3179a4 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:24.972980 9946 ts_tablet_manager.cc:1367] T 61dfb1d40ee444368470a0dafc3179a4 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:24.973304 9945 tablet_bootstrap.cc:492] T f6053da8b5a64536b738465aa6a3db10 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:24.973389 9945 ts_tablet_manager.cc:1367] T f6053da8b5a64536b738465aa6a3db10 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:24.971122 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 61dfb1d40ee444368470a0dafc3179a4 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b4dfb7bd2b7f40e9ad8e6e5c28e9ee31]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:24.973479 9946 raft_consensus.cc:348] T 61dfb1d40ee444368470a0dafc3179a4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:24.973592 9946 raft_consensus.cc:374] T 61dfb1d40ee444368470a0dafc3179a4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:24.973624 9946 raft_consensus.cc:729] T 61dfb1d40ee444368470a0dafc3179a4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:24.973659 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61dfb1d40ee444368470a0dafc3179a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:24.973742 9946 consensus_queue.cc:260] T 61dfb1d40ee444368470a0dafc3179a4 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:24.973814 9945 raft_consensus.cc:348] T f6053da8b5a64536b738465aa6a3db10 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:24.973932 9945 raft_consensus.cc:374] T f6053da8b5a64536b738465aa6a3db10 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:24.973965 9945 raft_consensus.cc:729] T f6053da8b5a64536b738465aa6a3db10 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:24.974066 9946 ts_tablet_manager.cc:1398] T 61dfb1d40ee444368470a0dafc3179a4 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:24.974104 9945 consensus_queue.cc:260] T f6053da8b5a64536b738465aa6a3db10 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:24.974287 9942 tablet_bootstrap.cc:492] T f6053da8b5a64536b738465aa6a3db10 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:24.974344 9945 ts_tablet_manager.cc:1398] T f6053da8b5a64536b738465aa6a3db10 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:24.974363 9942 ts_tablet_manager.cc:1367] T f6053da8b5a64536b738465aa6a3db10 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:59:24.974889 9942 raft_consensus.cc:348] T f6053da8b5a64536b738465aa6a3db10 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:24.974995 9942 raft_consensus.cc:374] T f6053da8b5a64536b738465aa6a3db10 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:24.975024 9942 raft_consensus.cc:729] T f6053da8b5a64536b738465aa6a3db10 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:24.975142 9942 consensus_queue.cc:260] T f6053da8b5a64536b738465aa6a3db10 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:24.975432 9942 ts_tablet_manager.cc:1398] T f6053da8b5a64536b738465aa6a3db10 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:24.975600 9942 tablet_bootstrap.cc:492] T 62c77ec99cf443d2b7d48b12c8855883 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:24.976452 9949 tablet_bootstrap.cc:492] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:24.976500 9942 tablet_bootstrap.cc:654] T 62c77ec99cf443d2b7d48b12c8855883 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:24.977342 9949 tablet_bootstrap.cc:654] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:24.977609 9942 tablet_bootstrap.cc:492] T 62c77ec99cf443d2b7d48b12c8855883 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:24.977694 9942 ts_tablet_manager.cc:1367] T 62c77ec99cf443d2b7d48b12c8855883 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:24.977885 9950 tablet_bootstrap.cc:492] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:24.978217 9942 raft_consensus.cc:348] T 62c77ec99cf443d2b7d48b12c8855883 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:24.978359 9942 raft_consensus.cc:374] T 62c77ec99cf443d2b7d48b12c8855883 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:24.978423 9942 raft_consensus.cc:729] T 62c77ec99cf443d2b7d48b12c8855883 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:24.978580 9942 consensus_queue.cc:260] T 62c77ec99cf443d2b7d48b12c8855883 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:24.978929 9942 ts_tablet_manager.cc:1398] T 62c77ec99cf443d2b7d48b12c8855883 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:24.979012 9943 tablet_bootstrap.cc:492] T f6053da8b5a64536b738465aa6a3db10 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:24.979454 9950 tablet_bootstrap.cc:654] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:24.979846 9943 tablet_bootstrap.cc:654] T f6053da8b5a64536b738465aa6a3db10 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:24.980698 9949 tablet_bootstrap.cc:492] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:24.980809 9949 ts_tablet_manager.cc:1367] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:59:24.981264 9949 raft_consensus.cc:348] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:24.981410 9949 raft_consensus.cc:374] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:24.981479 9949 raft_consensus.cc:729] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:24.981638 9943 tablet_bootstrap.cc:492] T f6053da8b5a64536b738465aa6a3db10 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:24.981632 9949 consensus_queue.cc:260] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:24.981820 9943 ts_tablet_manager.cc:1367] T f6053da8b5a64536b738465aa6a3db10 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:24.982004 9949 ts_tablet_manager.cc:1398] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:24.982196 9943 raft_consensus.cc:348] T f6053da8b5a64536b738465aa6a3db10 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:24.982283 9943 raft_consensus.cc:374] T f6053da8b5a64536b738465aa6a3db10 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:24.982329 9943 raft_consensus.cc:729] T f6053da8b5a64536b738465aa6a3db10 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:24.982452 9943 consensus_queue.cc:260] T f6053da8b5a64536b738465aa6a3db10 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:24.982865 9943 ts_tablet_manager.cc:1398] T f6053da8b5a64536b738465aa6a3db10 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:24.983062 9943 tablet_bootstrap.cc:492] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:24.983997 9943 tablet_bootstrap.cc:654] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:24.984999 9943 tablet_bootstrap.cc:492] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:24.985090 9943 ts_tablet_manager.cc:1367] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:24.985423 9943 raft_consensus.cc:348] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:24.985513 9943 raft_consensus.cc:374] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:24.985545 9943 raft_consensus.cc:729] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:24.985700 9943 consensus_queue.cc:260] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:24.986016 9943 ts_tablet_manager.cc:1398] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:24.987115 9950 tablet_bootstrap.cc:492] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:24.987221 9950 ts_tablet_manager.cc:1367] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:59:24.987666 9950 raft_consensus.cc:348] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:24.987785 9950 raft_consensus.cc:374] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:24.987828 9950 raft_consensus.cc:729] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:24.987977 9950 consensus_queue.cc:260] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:24.988284 9950 ts_tablet_manager.cc:1398] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:24.988464 9950 tablet_bootstrap.cc:492] T 61dfb1d40ee444368470a0dafc3179a4 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:24.989276 9950 tablet_bootstrap.cc:654] T 61dfb1d40ee444368470a0dafc3179a4 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:24.990520 9950 tablet_bootstrap.cc:492] T 61dfb1d40ee444368470a0dafc3179a4 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:24.990612 9950 ts_tablet_manager.cc:1367] T 61dfb1d40ee444368470a0dafc3179a4 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:24.990921 9950 raft_consensus.cc:348] T 61dfb1d40ee444368470a0dafc3179a4 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:24.990999 9950 raft_consensus.cc:374] T 61dfb1d40ee444368470a0dafc3179a4 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:24.991020 9950 raft_consensus.cc:729] T 61dfb1d40ee444368470a0dafc3179a4 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:24.991178 9950 consensus_queue.cc:260] T 61dfb1d40ee444368470a0dafc3179a4 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:24.991530 9950 ts_tablet_manager.cc:1398] T 61dfb1d40ee444368470a0dafc3179a4 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:25.002168 6383 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:25.002313 6383 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:25.004525 6383 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet df050fcf850345e38595f3312a22f60a
I20240503 13:59:25.005009 9190 tablet_service.cc:1514] Processing DeleteTablet for tablet df050fcf850345e38595f3312a22f60a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:24 UTC) from {username='slave'} at 127.0.0.1:57960
I20240503 13:59:25.005327 9957 tablet_replica.cc:330] T df050fcf850345e38595f3312a22f60a P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 13:59:25.005326 9323 tablet_service.cc:1514] Processing DeleteTablet for tablet df050fcf850345e38595f3312a22f60a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:24 UTC) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:25.005491 9957 raft_consensus.cc:2229] T df050fcf850345e38595f3312a22f60a P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:25.005599 9958 tablet_replica.cc:330] T df050fcf850345e38595f3312a22f60a P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:25.005607 6771 tablet_service.cc:1514] Processing DeleteTablet for tablet df050fcf850345e38595f3312a22f60a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:24 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:25.005702 9957 raft_consensus.cc:2258] T df050fcf850345e38595f3312a22f60a P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:25.005774 9958 raft_consensus.cc:2229] T df050fcf850345e38595f3312a22f60a P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:25.005877 9958 raft_consensus.cc:2258] T df050fcf850345e38595f3312a22f60a P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:25.006546 9959 tablet_replica.cc:330] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:25.006592 9958 ts_tablet_manager.cc:1875] T df050fcf850345e38595f3312a22f60a P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:25.006729 9959 raft_consensus.cc:2229] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:25.007025 9959 raft_consensus.cc:2258] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:25.007642 9957 ts_tablet_manager.cc:1875] T df050fcf850345e38595f3312a22f60a P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:25.007638 9959 ts_tablet_manager.cc:1875] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:25.009667 9958 ts_tablet_manager.cc:1888] T df050fcf850345e38595f3312a22f60a P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:25.009780 9958 log.cc:1196] T df050fcf850345e38595f3312a22f60a P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/df050fcf850345e38595f3312a22f60a
I20240503 13:59:25.009799 9959 ts_tablet_manager.cc:1888] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:25.009791 9957 ts_tablet_manager.cc:1888] T df050fcf850345e38595f3312a22f60a P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:25.009868 9957 log.cc:1196] T df050fcf850345e38595f3312a22f60a P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/df050fcf850345e38595f3312a22f60a
I20240503 13:59:25.009981 9959 log.cc:1196] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/df050fcf850345e38595f3312a22f60a
I20240503 13:59:25.010385 9958 ts_tablet_manager.cc:1909] T df050fcf850345e38595f3312a22f60a P 0d77752181e944698e809e325e5937e6: Deleting consensus metadata
I20240503 13:59:25.010406 9959 ts_tablet_manager.cc:1909] T df050fcf850345e38595f3312a22f60a P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:25.010454 9957 ts_tablet_manager.cc:1909] T df050fcf850345e38595f3312a22f60a P ab1d21935d6049888eb0ceccc3c9d735: Deleting consensus metadata
I20240503 13:59:25.011010 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet df050fcf850345e38595f3312a22f60a (table rebalancer_dynamic_table_new_name [id=db91e8e19d8344f198ca8a7d418b5be2]) successfully deleted
I20240503 13:59:25.011099 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet df050fcf850345e38595f3312a22f60a (table rebalancer_dynamic_table_new_name [id=db91e8e19d8344f198ca8a7d418b5be2]) successfully deleted
I20240503 13:59:25.011260 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet df050fcf850345e38595f3312a22f60a (table rebalancer_dynamic_table_new_name [id=db91e8e19d8344f198ca8a7d418b5be2]) successfully deleted
I20240503 13:59:25.050740 9789 raft_consensus.cc:482] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:25.050873 9789 raft_consensus.cc:504] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:25.051432 9789 leader_election.cc:288] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:25.052129 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61dfb1d40ee444368470a0dafc3179a4" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:25.052269 9080 raft_consensus.cc:2454] T 61dfb1d40ee444368470a0dafc3179a4 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:25.052558 6595 leader_election.cc:302] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:25.052742 9789 raft_consensus.cc:2781] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:25.052798 9789 raft_consensus.cc:482] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:25.052839 9789 raft_consensus.cc:3037] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:25.053956 9789 raft_consensus.cc:504] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:25.054325 9789 leader_election.cc:288] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:25.054615 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61dfb1d40ee444368470a0dafc3179a4" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:25.054741 9080 raft_consensus.cc:3037] T 61dfb1d40ee444368470a0dafc3179a4 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:25.055640 9342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61dfb1d40ee444368470a0dafc3179a4" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:25.055757 9342 raft_consensus.cc:2454] T 61dfb1d40ee444368470a0dafc3179a4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:25.055907 9080 raft_consensus.cc:2454] T 61dfb1d40ee444368470a0dafc3179a4 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:25.056191 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61dfb1d40ee444368470a0dafc3179a4" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:25.056226 6595 leader_election.cc:302] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:25.056293 9343 raft_consensus.cc:3037] T 61dfb1d40ee444368470a0dafc3179a4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:25.056367 9789 raft_consensus.cc:2781] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:25.056531 9789 raft_consensus.cc:686] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 13:59:25.056697 9789 consensus_queue.cc:237] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:25.057153 9343 raft_consensus.cc:2454] T 61dfb1d40ee444368470a0dafc3179a4 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:25.058357 6383 catalog_manager.cc:5484] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:25.107340 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:25.112208 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 7dc8f6cd3cd7408f88c5b66ffbdd6b8f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0841858522904c7ea8d53e878e2643ff]), partition=
I20240503 13:59:25.112260 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 7dc8f6cd3cd7408f88c5b66ffbdd6b8f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0841858522904c7ea8d53e878e2643ff]), partition=
I20240503 13:59:25.112263 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 7dc8f6cd3cd7408f88c5b66ffbdd6b8f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0841858522904c7ea8d53e878e2643ff]), partition=
I20240503 13:59:25.112475 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dc8f6cd3cd7408f88c5b66ffbdd6b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:25.112512 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dc8f6cd3cd7408f88c5b66ffbdd6b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:25.112512 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dc8f6cd3cd7408f88c5b66ffbdd6b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:25.114491 9950 tablet_bootstrap.cc:492] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:25.114538 9945 tablet_bootstrap.cc:492] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:25.114538 9943 tablet_bootstrap.cc:492] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:25.115427 9945 tablet_bootstrap.cc:654] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:25.115512 9950 tablet_bootstrap.cc:654] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:25.115516 9943 tablet_bootstrap.cc:654] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:25.116850 9950 tablet_bootstrap.cc:492] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:25.116936 9950 ts_tablet_manager.cc:1367] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:25.117372 9950 raft_consensus.cc:348] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:25.117477 9950 raft_consensus.cc:374] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:25.117520 9950 raft_consensus.cc:729] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:25.117548 9945 tablet_bootstrap.cc:492] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:25.117619 9945 ts_tablet_manager.cc:1367] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:59:25.117648 9950 consensus_queue.cc:260] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:25.117925 9950 ts_tablet_manager.cc:1398] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:25.118033 9945 raft_consensus.cc:348] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:25.118124 9945 raft_consensus.cc:374] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:25.118153 9945 raft_consensus.cc:729] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:25.118265 9945 consensus_queue.cc:260] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:25.118556 9945 ts_tablet_manager.cc:1398] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:25.119160 9943 tablet_bootstrap.cc:492] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:25.119246 9943 ts_tablet_manager.cc:1367] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 13:59:25.119758 9943 raft_consensus.cc:348] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:25.119865 9943 raft_consensus.cc:374] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:25.119904 9943 raft_consensus.cc:729] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:25.120028 9943 consensus_queue.cc:260] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:25.120285 9943 ts_tablet_manager.cc:1398] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:25.136909 9922 raft_consensus.cc:482] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:25.137022 9922 raft_consensus.cc:504] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:25.137298 9901 raft_consensus.cc:482] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:25.137394 9901 raft_consensus.cc:504] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:25.137461 9922 leader_election.cc:288] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:25.137771 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c77ec99cf443d2b7d48b12c8855883" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:25.137877 9901 leader_election.cc:288] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:25.137913 9205 raft_consensus.cc:2454] T 62c77ec99cf443d2b7d48b12c8855883 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:25.138526 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c77ec99cf443d2b7d48b12c8855883" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:25.138669 9205 raft_consensus.cc:2454] T 62c77ec99cf443d2b7d48b12c8855883 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:25.139393 9016 leader_election.cc:302] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:25.139600 9922 raft_consensus.cc:2781] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:25.139657 9922 raft_consensus.cc:482] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:25.139707 9922 raft_consensus.cc:3037] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:25.140882 9922 raft_consensus.cc:504] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:25.141152 9922 leader_election.cc:288] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:25.141525 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c77ec99cf443d2b7d48b12c8855883" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:25.141633 9205 raft_consensus.cc:3037] T 62c77ec99cf443d2b7d48b12c8855883 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:25.141662 6464 leader_election.cc:302] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:25.141822 9901 raft_consensus.cc:2781] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:25.141868 9901 raft_consensus.cc:482] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:25.141902 9901 raft_consensus.cc:3037] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:25.142613 9205 raft_consensus.cc:2454] T 62c77ec99cf443d2b7d48b12c8855883 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:25.142930 9901 raft_consensus.cc:504] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:25.143250 9901 leader_election.cc:288] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:25.143604 9016 leader_election.cc:302] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:25.143656 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c77ec99cf443d2b7d48b12c8855883" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:25.143776 6528 raft_consensus.cc:2379] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8d574fb4a89d4eea947c859470d44dd4 in current term 1: Already voted for candidate 5f46e19644fd4f4b825c86ba6af768da in this term.
I20240503 13:59:25.143837 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c77ec99cf443d2b7d48b12c8855883" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:25.143828 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c77ec99cf443d2b7d48b12c8855883" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:25.143959 6527 raft_consensus.cc:2379] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8d574fb4a89d4eea947c859470d44dd4 in current term 1: Already voted for candidate 5f46e19644fd4f4b825c86ba6af768da in this term.
I20240503 13:59:25.143977 9081 raft_consensus.cc:2379] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f46e19644fd4f4b825c86ba6af768da in current term 1: Already voted for candidate 8d574fb4a89d4eea947c859470d44dd4 in this term.
I20240503 13:59:25.144297 9922 raft_consensus.cc:2781] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:25.144330 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c77ec99cf443d2b7d48b12c8855883" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:25.144439 9922 raft_consensus.cc:686] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 13:59:25.144577 9922 consensus_queue.cc:237] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:25.144676 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c77ec99cf443d2b7d48b12c8855883" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:25.144793 9205 raft_consensus.cc:2379] T 62c77ec99cf443d2b7d48b12c8855883 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f46e19644fd4f4b825c86ba6af768da in current term 1: Already voted for candidate 8d574fb4a89d4eea947c859470d44dd4 in this term.
I20240503 13:59:25.145610 6464 leader_election.cc:302] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da; no voters: 8d574fb4a89d4eea947c859470d44dd4, ab1d21935d6049888eb0ceccc3c9d735
I20240503 13:59:25.145979 9901 raft_consensus.cc:2726] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 13:59:25.146390 6380 catalog_manager.cc:5484] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:25.149330 9901 raft_consensus.cc:482] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:25.149410 9901 raft_consensus.cc:504] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:25.149751 9901 leader_election.cc:288] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:25.150013 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc8f6cd3cd7408f88c5b66ffbdd6b8f" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:25.150131 9080 raft_consensus.cc:2454] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:25.150185 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc8f6cd3cd7408f88c5b66ffbdd6b8f" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:25.150322 8947 raft_consensus.cc:2454] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:25.150353 6464 leader_election.cc:302] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:25.150504 9901 raft_consensus.cc:2781] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:25.150553 9901 raft_consensus.cc:482] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:25.150601 9901 raft_consensus.cc:3037] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:25.151469 9901 raft_consensus.cc:504] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:25.151734 9901 leader_election.cc:288] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:25.151988 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc8f6cd3cd7408f88c5b66ffbdd6b8f" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:25.152112 9080 raft_consensus.cc:3037] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:25.152086 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc8f6cd3cd7408f88c5b66ffbdd6b8f" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:25.152199 8947 raft_consensus.cc:3037] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:25.152928 9080 raft_consensus.cc:2454] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:25.153118 8947 raft_consensus.cc:2454] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:25.153178 6464 leader_election.cc:302] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:25.153447 9901 raft_consensus.cc:2781] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:25.153661 9901 raft_consensus.cc:686] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Becoming Leader. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Running, Role: LEADER
I20240503 13:59:25.153816 9901 consensus_queue.cc:237] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:25.155577 6380 catalog_manager.cc:5484] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:25.215767 9828 raft_consensus.cc:482] T f6053da8b5a64536b738465aa6a3db10 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:25.215903 9828 raft_consensus.cc:504] T f6053da8b5a64536b738465aa6a3db10 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:25.216334 9828 leader_election.cc:288] T f6053da8b5a64536b738465aa6a3db10 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:25.216683 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6053da8b5a64536b738465aa6a3db10" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:25.216745 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6053da8b5a64536b738465aa6a3db10" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:25.216831 6527 raft_consensus.cc:2454] T f6053da8b5a64536b738465aa6a3db10 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:25.216858 8947 raft_consensus.cc:2454] T f6053da8b5a64536b738465aa6a3db10 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:25.217097 9147 leader_election.cc:302] T f6053da8b5a64536b738465aa6a3db10 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:25.217242 9828 raft_consensus.cc:2781] T f6053da8b5a64536b738465aa6a3db10 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:25.217314 9828 raft_consensus.cc:482] T f6053da8b5a64536b738465aa6a3db10 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:25.217360 9828 raft_consensus.cc:3037] T f6053da8b5a64536b738465aa6a3db10 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:25.218221 9828 raft_consensus.cc:504] T f6053da8b5a64536b738465aa6a3db10 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:25.218528 9828 leader_election.cc:288] T f6053da8b5a64536b738465aa6a3db10 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:25.218854 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6053da8b5a64536b738465aa6a3db10" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:25.218917 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6053da8b5a64536b738465aa6a3db10" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:25.218969 6527 raft_consensus.cc:3037] T f6053da8b5a64536b738465aa6a3db10 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:25.219019 8947 raft_consensus.cc:3037] T f6053da8b5a64536b738465aa6a3db10 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:25.219965 6527 raft_consensus.cc:2454] T f6053da8b5a64536b738465aa6a3db10 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:25.219965 8947 raft_consensus.cc:2454] T f6053da8b5a64536b738465aa6a3db10 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:25.220240 9147 leader_election.cc:302] T f6053da8b5a64536b738465aa6a3db10 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:25.220392 9828 raft_consensus.cc:2781] T f6053da8b5a64536b738465aa6a3db10 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:25.220466 9828 raft_consensus.cc:686] T f6053da8b5a64536b738465aa6a3db10 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 13:59:25.220588 9828 consensus_queue.cc:237] T f6053da8b5a64536b738465aa6a3db10 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:25.222450 6380 catalog_manager.cc:5484] T f6053da8b5a64536b738465aa6a3db10 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:25.262882 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:25.266671 8947 raft_consensus.cc:1261] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:25.266748 9080 raft_consensus.cc:1261] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:25.267036 9973 consensus_queue.cc:1035] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:25.267189 9901 consensus_queue.cc:1035] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:25.269377 6380 catalog_manager.cc:5576] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481) does not have the latest schema for tablet 7dc8f6cd3cd7408f88c5b66ffbdd6b8f (table rebalancer_dynamic_table [id=0841858522904c7ea8d53e878e2643ff]). Expected version 1 got 0
I20240503 13:59:25.269714 6385 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet 7dc8f6cd3cd7408f88c5b66ffbdd6b8f (table rebalancer_dynamic_table [id=0841858522904c7ea8d53e878e2643ff]). Expected version 1 got 0
I20240503 13:59:25.271523 9980 tablet.cc:1722] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:25.271587 9979 tablet.cc:1722] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:25.271761 6385 catalog_manager.cc:5576] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481) does not have the latest schema for tablet 7dc8f6cd3cd7408f88c5b66ffbdd6b8f (table rebalancer_dynamic_table [id=0841858522904c7ea8d53e878e2643ff]). Expected version 1 got 0
I20240503 13:59:25.271852 9979 tablet.cc:1620] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:25.272020 9980 tablet.cc:1620] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da: MemRowSet was empty: no flush needed.
I20240503 13:59:25.273686 9981 tablet.cc:1722] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:25.274179 9981 tablet.cc:1620] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:25.275168 6380 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet 7dc8f6cd3cd7408f88c5b66ffbdd6b8f (table rebalancer_dynamic_table [id=0841858522904c7ea8d53e878e2643ff]). Expected version 1 got 0
I20240503 13:59:25.278159 6385 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=0841858522904c7ea8d53e878e2643ff] alter complete (version 1)
I20240503 13:59:25.385285 6385 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:25.390995 9979 tablet.cc:1722] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:25.391016 9980 tablet.cc:1722] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:25.391392 9979 tablet.cc:1620] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:25.391649 9980 tablet.cc:1620] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da: MemRowSet was empty: no flush needed.
I20240503 13:59:25.391841 9981 tablet.cc:1722] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:25.392131 9981 tablet.cc:1620] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:25.396862 6385 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=0841858522904c7ea8d53e878e2643ff] alter complete (version 2)
I20240503 13:59:25.501237 6385 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:25.504729 9962 consensus_queue.cc:1035] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:25.507750 9979 tablet.cc:1722] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:25.508106 9981 tablet.cc:1722] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:25.508122 9962 consensus_queue.cc:1035] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:25.508929 9980 tablet.cc:1722] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:25.513289 6385 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=0841858522904c7ea8d53e878e2643ff] alter complete (version 3)
I20240503 13:59:25.554144 9922 consensus_queue.cc:1035] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:25.557171 9987 consensus_queue.cc:1035] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:25.614923 6385 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:25.615095 6385 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:25.617205 6385 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 7dc8f6cd3cd7408f88c5b66ffbdd6b8f
I20240503 13:59:25.617712 9058 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dc8f6cd3cd7408f88c5b66ffbdd6b8f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:25 UTC) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:25.617806 8928 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dc8f6cd3cd7408f88c5b66ffbdd6b8f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:25 UTC) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:25.617916 6508 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dc8f6cd3cd7408f88c5b66ffbdd6b8f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:25 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:25.618134 9990 tablet_replica.cc:330] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:25.618147 9991 tablet_replica.cc:330] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:25.618264 9990 raft_consensus.cc:2229] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:25.618294 9991 raft_consensus.cc:2229] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:25.618383 9990 raft_consensus.cc:2258] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:25.618398 9992 tablet_replica.cc:330] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:25.618453 9991 raft_consensus.cc:2258] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:25.618773 9992 raft_consensus.cc:2229] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:25.618973 9990 ts_tablet_manager.cc:1875] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:25.618997 9992 raft_consensus.cc:2258] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:25.619019 9991 ts_tablet_manager.cc:1875] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:25.619537 9992 ts_tablet_manager.cc:1875] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:25.620918 9990 ts_tablet_manager.cc:1888] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:25.621070 9990 log.cc:1196] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/7dc8f6cd3cd7408f88c5b66ffbdd6b8f
I20240503 13:59:25.621325 9990 ts_tablet_manager.cc:1909] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 8d574fb4a89d4eea947c859470d44dd4: Deleting consensus metadata
I20240503 13:59:25.621415 9991 ts_tablet_manager.cc:1888] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:25.621542 9991 log.cc:1196] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/7dc8f6cd3cd7408f88c5b66ffbdd6b8f
I20240503 13:59:25.621807 9991 ts_tablet_manager.cc:1909] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 06ca8dbc004c42b99faa527a4e6f9545: Deleting consensus metadata
I20240503 13:59:25.621991 9992 ts_tablet_manager.cc:1888] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:25.622087 9992 log.cc:1196] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/7dc8f6cd3cd7408f88c5b66ffbdd6b8f
I20240503 13:59:25.622058 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet 7dc8f6cd3cd7408f88c5b66ffbdd6b8f (table rebalancer_dynamic_table_new_name [id=0841858522904c7ea8d53e878e2643ff]) successfully deleted
I20240503 13:59:25.622403 9992 ts_tablet_manager.cc:1909] T 7dc8f6cd3cd7408f88c5b66ffbdd6b8f P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:25.623175 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 7dc8f6cd3cd7408f88c5b66ffbdd6b8f (table rebalancer_dynamic_table_new_name [id=0841858522904c7ea8d53e878e2643ff]) successfully deleted
I20240503 13:59:25.623327 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 7dc8f6cd3cd7408f88c5b66ffbdd6b8f (table rebalancer_dynamic_table_new_name [id=0841858522904c7ea8d53e878e2643ff]) successfully deleted
I20240503 13:59:25.718796 6385 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:25.723493 9323 tablet_service.cc:1467] Processing CreateTablet for tablet 2e0d48ba04ea44aa9c7656e79917a2b5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=470c00feaeac4f028c10b764757c8778]), partition=
I20240503 13:59:25.723703 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 2e0d48ba04ea44aa9c7656e79917a2b5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=470c00feaeac4f028c10b764757c8778]), partition=
I20240503 13:59:25.723718 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 2e0d48ba04ea44aa9c7656e79917a2b5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=470c00feaeac4f028c10b764757c8778]), partition=
I20240503 13:59:25.723798 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e0d48ba04ea44aa9c7656e79917a2b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:25.723951 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e0d48ba04ea44aa9c7656e79917a2b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:25.723958 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e0d48ba04ea44aa9c7656e79917a2b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:25.725592 9995 tablet_bootstrap.cc:492] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:25.726272 9996 tablet_bootstrap.cc:492] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:25.726272 9997 tablet_bootstrap.cc:492] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:25.726799 9995 tablet_bootstrap.cc:654] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:25.727095 9997 tablet_bootstrap.cc:654] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:25.727193 9996 tablet_bootstrap.cc:654] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:25.727809 9995 tablet_bootstrap.cc:492] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:25.727902 9995 ts_tablet_manager.cc:1367] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:25.728199 9997 tablet_bootstrap.cc:492] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:25.728291 9997 ts_tablet_manager.cc:1367] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:25.728322 9995 raft_consensus.cc:348] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:25.728452 9995 raft_consensus.cc:374] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:25.728482 9996 tablet_bootstrap.cc:492] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:25.728508 9995 raft_consensus.cc:729] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:25.728631 9996 ts_tablet_manager.cc:1367] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:25.728678 9995 consensus_queue.cc:260] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:25.728782 9997 raft_consensus.cc:348] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:25.728905 9997 raft_consensus.cc:374] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:25.728956 9997 raft_consensus.cc:729] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:25.729094 9996 raft_consensus.cc:348] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:25.729104 9997 consensus_queue.cc:260] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:25.729211 9996 raft_consensus.cc:374] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:25.729287 9996 raft_consensus.cc:729] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:25.729460 9996 consensus_queue.cc:260] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:25.729730 9997 ts_tablet_manager.cc:1398] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:25.730085 9995 ts_tablet_manager.cc:1398] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:25.730239 9996 ts_tablet_manager.cc:1398] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:25.735540 9828 consensus_queue.cc:1035] T f6053da8b5a64536b738465aa6a3db10 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:25.736563 9828 consensus_queue.cc:1035] T f6053da8b5a64536b738465aa6a3db10 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:25.764063 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:25.770409 8928 tablet_service.cc:1467] Processing CreateTablet for tablet d51634c08e2749d4a3bbff4b2a786e48 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=82c39415abdc4fe8a6c8d9f312c84cbc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:25.770471 6901 tablet_service.cc:1467] Processing CreateTablet for tablet d51634c08e2749d4a3bbff4b2a786e48 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=82c39415abdc4fe8a6c8d9f312c84cbc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:25.770471 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 0fa4727297da40eb9f58f5daa8150e0e (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=82c39415abdc4fe8a6c8d9f312c84cbc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:25.770728 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d51634c08e2749d4a3bbff4b2a786e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:25.770747 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fa4727297da40eb9f58f5daa8150e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:25.770747 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d51634c08e2749d4a3bbff4b2a786e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:25.770769 9058 tablet_service.cc:1467] Processing CreateTablet for tablet d51634c08e2749d4a3bbff4b2a786e48 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=82c39415abdc4fe8a6c8d9f312c84cbc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:25.771006 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d51634c08e2749d4a3bbff4b2a786e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:25.773279 9995 tablet_bootstrap.cc:492] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:25.773746 10007 tablet_bootstrap.cc:492] T d51634c08e2749d4a3bbff4b2a786e48 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:25.773869 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 0fa4727297da40eb9f58f5daa8150e0e (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=82c39415abdc4fe8a6c8d9f312c84cbc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:25.774081 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fa4727297da40eb9f58f5daa8150e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:25.774039 9323 tablet_service.cc:1467] Processing CreateTablet for tablet 0fa4727297da40eb9f58f5daa8150e0e (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=82c39415abdc4fe8a6c8d9f312c84cbc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:25.774299 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fa4727297da40eb9f58f5daa8150e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:25.774392 9995 tablet_bootstrap.cc:654] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:25.774747 10007 tablet_bootstrap.cc:654] T d51634c08e2749d4a3bbff4b2a786e48 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:25.775527 6771 tablet_service.cc:1467] Processing CreateTablet for tablet 6321c633e38e4c9e853705b2b271326f (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=82c39415abdc4fe8a6c8d9f312c84cbc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:25.775736 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6321c633e38e4c9e853705b2b271326f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:25.776422 9995 tablet_bootstrap.cc:492] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:25.776455 9996 tablet_bootstrap.cc:492] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:25.776494 9995 ts_tablet_manager.cc:1367] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:25.776635 8927 tablet_service.cc:1467] Processing CreateTablet for tablet 6321c633e38e4c9e853705b2b271326f (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=82c39415abdc4fe8a6c8d9f312c84cbc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:25.776839 8927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6321c633e38e4c9e853705b2b271326f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:25.777130 9995 raft_consensus.cc:348] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:25.777228 9995 raft_consensus.cc:374] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:25.777263 9995 raft_consensus.cc:729] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:25.777381 9995 consensus_queue.cc:260] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:25.777429 9996 tablet_bootstrap.cc:654] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:25.777674 9995 ts_tablet_manager.cc:1398] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:25.777812 9995 tablet_bootstrap.cc:492] T 0fa4727297da40eb9f58f5daa8150e0e P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:25.778668 9995 tablet_bootstrap.cc:654] T 0fa4727297da40eb9f58f5daa8150e0e P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:25.779413 10005 tablet_bootstrap.cc:492] T d51634c08e2749d4a3bbff4b2a786e48 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:25.780392 10012 tablet_bootstrap.cc:492] T 6321c633e38e4c9e853705b2b271326f P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:25.780500 10006 tablet_bootstrap.cc:492] T 0fa4727297da40eb9f58f5daa8150e0e P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:25.780718 10005 tablet_bootstrap.cc:654] T d51634c08e2749d4a3bbff4b2a786e48 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:25.780918 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 6321c633e38e4c9e853705b2b271326f (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=82c39415abdc4fe8a6c8d9f312c84cbc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:25.781131 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6321c633e38e4c9e853705b2b271326f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:25.781461 10012 tablet_bootstrap.cc:654] T 6321c633e38e4c9e853705b2b271326f P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:25.781648 10006 tablet_bootstrap.cc:654] T 0fa4727297da40eb9f58f5daa8150e0e P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:25.782861 10006 tablet_bootstrap.cc:492] T 0fa4727297da40eb9f58f5daa8150e0e P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:25.782864 10012 tablet_bootstrap.cc:492] T 6321c633e38e4c9e853705b2b271326f P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:25.782922 10006 ts_tablet_manager.cc:1367] T 0fa4727297da40eb9f58f5daa8150e0e P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:25.782940 10012 ts_tablet_manager.cc:1367] T 6321c633e38e4c9e853705b2b271326f P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:25.783375 10006 raft_consensus.cc:348] T 0fa4727297da40eb9f58f5daa8150e0e P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:25.783388 10012 raft_consensus.cc:348] T 6321c633e38e4c9e853705b2b271326f P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:25.783471 10006 raft_consensus.cc:374] T 0fa4727297da40eb9f58f5daa8150e0e P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:25.783483 10012 raft_consensus.cc:374] T 6321c633e38e4c9e853705b2b271326f P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:25.783514 10012 raft_consensus.cc:729] T 6321c633e38e4c9e853705b2b271326f P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:25.783532 10006 raft_consensus.cc:729] T 0fa4727297da40eb9f58f5daa8150e0e P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:25.783635 10012 consensus_queue.cc:260] T 6321c633e38e4c9e853705b2b271326f P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:25.783672 10006 consensus_queue.cc:260] T 0fa4727297da40eb9f58f5daa8150e0e P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:25.783942 10012 ts_tablet_manager.cc:1398] T 6321c633e38e4c9e853705b2b271326f P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:25.784446 10006 ts_tablet_manager.cc:1398] T 0fa4727297da40eb9f58f5daa8150e0e P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:25.784646 10007 tablet_bootstrap.cc:492] T d51634c08e2749d4a3bbff4b2a786e48 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:25.784750 10007 ts_tablet_manager.cc:1367] T d51634c08e2749d4a3bbff4b2a786e48 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 13:59:25.784921 9995 tablet_bootstrap.cc:492] T 0fa4727297da40eb9f58f5daa8150e0e P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:25.785004 9995 ts_tablet_manager.cc:1367] T 0fa4727297da40eb9f58f5daa8150e0e P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 13:59:25.785135 9996 tablet_bootstrap.cc:492] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:25.785218 9996 ts_tablet_manager.cc:1367] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:59:25.785220 10007 raft_consensus.cc:348] T d51634c08e2749d4a3bbff4b2a786e48 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:25.785317 10007 raft_consensus.cc:374] T d51634c08e2749d4a3bbff4b2a786e48 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:25.785353 10007 raft_consensus.cc:729] T d51634c08e2749d4a3bbff4b2a786e48 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:25.785465 10007 consensus_queue.cc:260] T d51634c08e2749d4a3bbff4b2a786e48 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:25.785455 9995 raft_consensus.cc:348] T 0fa4727297da40eb9f58f5daa8150e0e P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:25.785558 9995 raft_consensus.cc:374] T 0fa4727297da40eb9f58f5daa8150e0e P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:25.785596 9995 raft_consensus.cc:729] T 0fa4727297da40eb9f58f5daa8150e0e P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:25.785684 9996 raft_consensus.cc:348] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:25.785751 10007 ts_tablet_manager.cc:1398] T d51634c08e2749d4a3bbff4b2a786e48 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:25.785709 9995 consensus_queue.cc:260] T 0fa4727297da40eb9f58f5daa8150e0e P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:25.785786 9996 raft_consensus.cc:374] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:25.785817 9996 raft_consensus.cc:729] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:25.785912 10007 tablet_bootstrap.cc:492] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:25.785933 9996 consensus_queue.cc:260] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:25.785989 9995 ts_tablet_manager.cc:1398] T 0fa4727297da40eb9f58f5daa8150e0e P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:25.786137 9995 tablet_bootstrap.cc:492] T 6321c633e38e4c9e853705b2b271326f P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:25.786186 9996 ts_tablet_manager.cc:1398] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:25.786909 10007 tablet_bootstrap.cc:654] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:25.787057 9995 tablet_bootstrap.cc:654] T 6321c633e38e4c9e853705b2b271326f P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:25.788308 10005 tablet_bootstrap.cc:492] T d51634c08e2749d4a3bbff4b2a786e48 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:25.788445 10005 ts_tablet_manager.cc:1367] T d51634c08e2749d4a3bbff4b2a786e48 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:59:25.788821 10005 raft_consensus.cc:348] T d51634c08e2749d4a3bbff4b2a786e48 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:25.788904 10005 raft_consensus.cc:374] T d51634c08e2749d4a3bbff4b2a786e48 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:25.788924 10005 raft_consensus.cc:729] T d51634c08e2749d4a3bbff4b2a786e48 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:25.789029 10005 consensus_queue.cc:260] T d51634c08e2749d4a3bbff4b2a786e48 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:25.789302 10005 ts_tablet_manager.cc:1398] T d51634c08e2749d4a3bbff4b2a786e48 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:25.793597 9995 tablet_bootstrap.cc:492] T 6321c633e38e4c9e853705b2b271326f P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:25.793596 10007 tablet_bootstrap.cc:492] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:25.793684 10007 ts_tablet_manager.cc:1367] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:59:25.793684 9995 ts_tablet_manager.cc:1367] T 6321c633e38e4c9e853705b2b271326f P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:59:25.794111 10007 raft_consensus.cc:348] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:25.794102 9995 raft_consensus.cc:348] T 6321c633e38e4c9e853705b2b271326f P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:25.794214 10007 raft_consensus.cc:374] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:25.794214 9995 raft_consensus.cc:374] T 6321c633e38e4c9e853705b2b271326f P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:25.794261 9995 raft_consensus.cc:729] T 6321c633e38e4c9e853705b2b271326f P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:25.794261 10007 raft_consensus.cc:729] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:25.794384 9995 consensus_queue.cc:260] T 6321c633e38e4c9e853705b2b271326f P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:25.794384 10007 consensus_queue.cc:260] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:25.794648 9995 ts_tablet_manager.cc:1398] T 6321c633e38e4c9e853705b2b271326f P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:25.794649 10007 ts_tablet_manager.cc:1398] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:25.794868 9994 raft_consensus.cc:482] T 6321c633e38e4c9e853705b2b271326f P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:25.794960 9994 raft_consensus.cc:504] T 6321c633e38e4c9e853705b2b271326f P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:25.795518 9994 leader_election.cc:288] T 6321c633e38e4c9e853705b2b271326f P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:25.800287 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6321c633e38e4c9e853705b2b271326f" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:25.800443 8947 raft_consensus.cc:2454] T 6321c633e38e4c9e853705b2b271326f P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:25.800570 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6321c633e38e4c9e853705b2b271326f" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:25.800707 9081 raft_consensus.cc:2454] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:25.800915 6726 leader_election.cc:302] T 6321c633e38e4c9e853705b2b271326f P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:25.801199 9994 raft_consensus.cc:2781] T 6321c633e38e4c9e853705b2b271326f P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:25.801259 9994 raft_consensus.cc:482] T 6321c633e38e4c9e853705b2b271326f P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:25.801294 9994 raft_consensus.cc:3037] T 6321c633e38e4c9e853705b2b271326f P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:25.802326 9994 raft_consensus.cc:504] T 6321c633e38e4c9e853705b2b271326f P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:25.802629 9994 leader_election.cc:288] T 6321c633e38e4c9e853705b2b271326f P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:25.802904 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6321c633e38e4c9e853705b2b271326f" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:25.802996 8947 raft_consensus.cc:3037] T 6321c633e38e4c9e853705b2b271326f P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:25.803012 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6321c633e38e4c9e853705b2b271326f" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:25.803123 9081 raft_consensus.cc:3037] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:25.803831 8947 raft_consensus.cc:2454] T 6321c633e38e4c9e853705b2b271326f P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:25.803869 9081 raft_consensus.cc:2454] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:25.804080 6726 leader_election.cc:302] T 6321c633e38e4c9e853705b2b271326f P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:25.804230 9994 raft_consensus.cc:2781] T 6321c633e38e4c9e853705b2b271326f P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:25.804329 9994 raft_consensus.cc:686] T 6321c633e38e4c9e853705b2b271326f P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:25.804472 9994 consensus_queue.cc:237] T 6321c633e38e4c9e853705b2b271326f P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:25.806434 6385 catalog_manager.cc:5484] T 6321c633e38e4c9e853705b2b271326f P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:25.808187 9923 raft_consensus.cc:482] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:25.808269 9923 raft_consensus.cc:504] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:25.808637 9923 leader_election.cc:288] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:25.809019 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0d48ba04ea44aa9c7656e79917a2b5" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:25.809168 9343 raft_consensus.cc:2454] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:25.809424 8884 leader_election.cc:302] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 0d77752181e944698e809e325e5937e6; no voters:
I20240503 13:59:25.809401 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0d48ba04ea44aa9c7656e79917a2b5" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:25.809532 6659 raft_consensus.cc:2454] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:25.809583 9923 raft_consensus.cc:2781] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:25.809648 9923 raft_consensus.cc:482] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:25.809692 9923 raft_consensus.cc:3037] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:25.810714 9923 raft_consensus.cc:504] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:25.811046 9923 leader_election.cc:288] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:25.811371 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0d48ba04ea44aa9c7656e79917a2b5" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:25.811430 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0d48ba04ea44aa9c7656e79917a2b5" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:25.811491 9343 raft_consensus.cc:3037] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:25.811537 6659 raft_consensus.cc:3037] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:25.812297 6659 raft_consensus.cc:2454] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:25.812366 9343 raft_consensus.cc:2454] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:25.812577 8884 leader_election.cc:302] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 2145d15ab2d74c0287839266f1411a43; no voters:
I20240503 13:59:25.812727 9923 raft_consensus.cc:2781] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:25.812808 9923 raft_consensus.cc:686] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 13:59:25.812939 9923 consensus_queue.cc:237] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:25.815343 6383 catalog_manager.cc:5484] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:25.928797 6383 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:25.933683 9343 raft_consensus.cc:1261] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:25.934087 6659 raft_consensus.cc:1261] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:25.934320 9923 consensus_queue.cc:1035] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:25.934459 9717 consensus_queue.cc:1035] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:25.936978 6383 catalog_manager.cc:5576] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931) does not have the latest schema for tablet 2e0d48ba04ea44aa9c7656e79917a2b5 (table rebalancer_dynamic_table [id=470c00feaeac4f028c10b764757c8778]). Expected version 1 got 0
I20240503 13:59:25.937775 6380 catalog_manager.cc:5576] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119) does not have the latest schema for tablet 2e0d48ba04ea44aa9c7656e79917a2b5 (table rebalancer_dynamic_table [id=470c00feaeac4f028c10b764757c8778]). Expected version 1 got 0
I20240503 13:59:25.938004 9981 tablet.cc:1722] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:25.938318 9981 tablet.cc:1620] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:25.940047 10029 tablet.cc:1722] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:25.940375 10029 tablet.cc:1620] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:25.940651 6380 catalog_manager.cc:5576] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119) does not have the latest schema for tablet 2e0d48ba04ea44aa9c7656e79917a2b5 (table rebalancer_dynamic_table [id=470c00feaeac4f028c10b764757c8778]). Expected version 1 got 0
I20240503 13:59:25.941094 6385 catalog_manager.cc:5576] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931) does not have the latest schema for tablet 2e0d48ba04ea44aa9c7656e79917a2b5 (table rebalancer_dynamic_table [id=470c00feaeac4f028c10b764757c8778]). Expected version 1 got 0
I20240503 13:59:25.943461 10028 tablet.cc:1722] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:25.943786 10028 tablet.cc:1620] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 2145d15ab2d74c0287839266f1411a43: MemRowSet was empty: no flush needed.
I20240503 13:59:25.945734 6383 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=470c00feaeac4f028c10b764757c8778] alter complete (version 1)
I20240503 13:59:26.049731 6385 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:26.054627 9981 tablet.cc:1722] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:26.054953 9981 tablet.cc:1620] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:26.055765 10029 tablet.cc:1722] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:26.056001 10028 tablet.cc:1722] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:26.056102 10029 tablet.cc:1620] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:26.056517 10028 tablet.cc:1620] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 2145d15ab2d74c0287839266f1411a43: MemRowSet was empty: no flush needed.
I20240503 13:59:26.058545 9937 raft_consensus.cc:482] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:26.058645 9937 raft_consensus.cc:504] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:26.058972 9937 leader_election.cc:288] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:26.059393 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fa4727297da40eb9f58f5daa8150e0e" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:26.059541 6527 raft_consensus.cc:2454] T 0fa4727297da40eb9f58f5daa8150e0e P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:26.059574 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fa4727297da40eb9f58f5daa8150e0e" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:26.059721 8947 raft_consensus.cc:2454] T 0fa4727297da40eb9f58f5daa8150e0e P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:26.060173 6385 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=470c00feaeac4f028c10b764757c8778] alter complete (version 2)
I20240503 13:59:26.061165 9278 leader_election.cc:302] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:26.061327 9937 raft_consensus.cc:2781] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:26.061383 9937 raft_consensus.cc:482] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:26.061424 9937 raft_consensus.cc:3037] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:26.062120 9937 raft_consensus.cc:504] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:26.062412 9937 leader_election.cc:288] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:26.062731 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fa4727297da40eb9f58f5daa8150e0e" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:26.062748 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fa4727297da40eb9f58f5daa8150e0e" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:26.062829 8947 raft_consensus.cc:3037] T 0fa4727297da40eb9f58f5daa8150e0e P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:26.062839 6527 raft_consensus.cc:3037] T 0fa4727297da40eb9f58f5daa8150e0e P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:26.063568 8947 raft_consensus.cc:2454] T 0fa4727297da40eb9f58f5daa8150e0e P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:26.063577 6527 raft_consensus.cc:2454] T 0fa4727297da40eb9f58f5daa8150e0e P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:26.063839 9278 leader_election.cc:302] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:26.063987 9937 raft_consensus.cc:2781] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:26.064085 9937 raft_consensus.cc:686] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:26.064193 9937 consensus_queue.cc:237] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:26.065656 6380 catalog_manager.cc:5484] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:26.074465 9717 raft_consensus.cc:482] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:26.074553 9717 raft_consensus.cc:504] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:26.074906 9717 leader_election.cc:288] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:26.075309 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d51634c08e2749d4a3bbff4b2a786e48" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:26.075454 9081 raft_consensus.cc:2454] T d51634c08e2749d4a3bbff4b2a786e48 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:26.075744 8883 leader_election.cc:302] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:26.075913 9717 raft_consensus.cc:2781] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:26.075978 9717 raft_consensus.cc:482] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:26.076021 9717 raft_consensus.cc:3037] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:26.076959 9717 raft_consensus.cc:504] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:26.077267 9717 leader_election.cc:288] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:26.077636 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d51634c08e2749d4a3bbff4b2a786e48" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:26.077772 9081 raft_consensus.cc:3037] T d51634c08e2749d4a3bbff4b2a786e48 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:26.078684 9081 raft_consensus.cc:2454] T d51634c08e2749d4a3bbff4b2a786e48 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:26.079059 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d51634c08e2749d4a3bbff4b2a786e48" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:26.079111 8883 leader_election.cc:302] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:26.079104 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d51634c08e2749d4a3bbff4b2a786e48" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:26.079195 6920 raft_consensus.cc:2454] T d51634c08e2749d4a3bbff4b2a786e48 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:26.079308 9717 raft_consensus.cc:2781] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:26.079384 9717 raft_consensus.cc:686] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 13:59:26.079514 9717 consensus_queue.cc:237] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:26.081044 6380 catalog_manager.cc:5484] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:26.165177 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:26.169976 9981 tablet.cc:1722] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:26.170508 10028 tablet.cc:1722] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:26.170588 10029 tablet.cc:1722] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:26.174226 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=470c00feaeac4f028c10b764757c8778] alter complete (version 3)
I20240503 13:59:26.237623 9994 consensus_queue.cc:1035] T 6321c633e38e4c9e853705b2b271326f P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:26.240613 9994 consensus_queue.cc:1035] T 6321c633e38e4c9e853705b2b271326f P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:26.277551 6383 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:26.277714 6383 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:26.279883 6383 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 2e0d48ba04ea44aa9c7656e79917a2b5
I20240503 13:59:26.280438 9323 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e0d48ba04ea44aa9c7656e79917a2b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:26 UTC) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:26.280494 8927 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e0d48ba04ea44aa9c7656e79917a2b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:26 UTC) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:26.280756 10040 tablet_replica.cc:330] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:26.280892 10041 tablet_replica.cc:330] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:26.281047 10041 raft_consensus.cc:2229] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:26.281087 6640 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e0d48ba04ea44aa9c7656e79917a2b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:26 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:26.281248 10041 raft_consensus.cc:2258] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:26.281365 10042 tablet_replica.cc:330] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:26.281526 10042 raft_consensus.cc:2229] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:26.281683 10042 raft_consensus.cc:2258] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:26.281900 10041 ts_tablet_manager.cc:1875] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:26.283038 10040 raft_consensus.cc:2229] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:26.283201 10040 raft_consensus.cc:2258] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:26.283313 10042 ts_tablet_manager.cc:1875] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:26.285305 10041 ts_tablet_manager.cc:1888] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:26.285378 10041 log.cc:1196] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/2e0d48ba04ea44aa9c7656e79917a2b5
I20240503 13:59:26.285512 10042 ts_tablet_manager.cc:1888] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:26.285583 10042 log.cc:1196] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/2e0d48ba04ea44aa9c7656e79917a2b5
I20240503 13:59:26.285641 10041 ts_tablet_manager.cc:1909] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting consensus metadata
I20240503 13:59:26.285874 10042 ts_tablet_manager.cc:1909] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:26.286399 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 2e0d48ba04ea44aa9c7656e79917a2b5 (table rebalancer_dynamic_table_new_name [id=470c00feaeac4f028c10b764757c8778]) successfully deleted
I20240503 13:59:26.286566 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet 2e0d48ba04ea44aa9c7656e79917a2b5 (table rebalancer_dynamic_table_new_name [id=470c00feaeac4f028c10b764757c8778]) successfully deleted
I20240503 13:59:26.287693 10040 ts_tablet_manager.cc:1875] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:26.289757 10040 ts_tablet_manager.cc:1888] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:26.289827 10040 log.cc:1196] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/2e0d48ba04ea44aa9c7656e79917a2b5
I20240503 13:59:26.290084 10040 ts_tablet_manager.cc:1909] T 2e0d48ba04ea44aa9c7656e79917a2b5 P 0d77752181e944698e809e325e5937e6: Deleting consensus metadata
I20240503 13:59:26.290712 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet 2e0d48ba04ea44aa9c7656e79917a2b5 (table rebalancer_dynamic_table_new_name [id=470c00feaeac4f028c10b764757c8778]) successfully deleted
I20240503 13:59:26.381807 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:26.386760 9323 tablet_service.cc:1467] Processing CreateTablet for tablet 518a2226d5754b80996ec134e448c7fb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5dd58c7354c64075ab0018a952bdc68a]), partition=
I20240503 13:59:26.386826 6771 tablet_service.cc:1467] Processing CreateTablet for tablet 518a2226d5754b80996ec134e448c7fb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5dd58c7354c64075ab0018a952bdc68a]), partition=
I20240503 13:59:26.386909 9190 tablet_service.cc:1467] Processing CreateTablet for tablet 518a2226d5754b80996ec134e448c7fb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5dd58c7354c64075ab0018a952bdc68a]), partition=
I20240503 13:59:26.387037 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 518a2226d5754b80996ec134e448c7fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:26.387089 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 518a2226d5754b80996ec134e448c7fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:26.387112 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 518a2226d5754b80996ec134e448c7fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:26.389398 10045 tablet_bootstrap.cc:492] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:26.389632 10043 tablet_bootstrap.cc:492] T 518a2226d5754b80996ec134e448c7fb P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:26.390278 10045 tablet_bootstrap.cc:654] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:26.390530 10043 tablet_bootstrap.cc:654] T 518a2226d5754b80996ec134e448c7fb P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:26.390888 10044 tablet_bootstrap.cc:492] T 518a2226d5754b80996ec134e448c7fb P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:26.391371 10045 tablet_bootstrap.cc:492] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:26.391458 10045 ts_tablet_manager.cc:1367] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:26.391568 10043 tablet_bootstrap.cc:492] T 518a2226d5754b80996ec134e448c7fb P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:26.391664 10043 ts_tablet_manager.cc:1367] T 518a2226d5754b80996ec134e448c7fb P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:26.391746 10044 tablet_bootstrap.cc:654] T 518a2226d5754b80996ec134e448c7fb P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:26.391912 10045 raft_consensus.cc:348] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:26.392012 10045 raft_consensus.cc:374] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:26.392071 10045 raft_consensus.cc:729] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:26.392189 10043 raft_consensus.cc:348] T 518a2226d5754b80996ec134e448c7fb P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:26.392226 10045 consensus_queue.cc:260] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:26.392313 10043 raft_consensus.cc:374] T 518a2226d5754b80996ec134e448c7fb P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:26.392356 10043 raft_consensus.cc:729] T 518a2226d5754b80996ec134e448c7fb P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:26.392469 10043 consensus_queue.cc:260] T 518a2226d5754b80996ec134e448c7fb P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:26.392624 10045 ts_tablet_manager.cc:1398] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:26.392714 10043 ts_tablet_manager.cc:1398] T 518a2226d5754b80996ec134e448c7fb P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:26.393692 10044 tablet_bootstrap.cc:492] T 518a2226d5754b80996ec134e448c7fb P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:26.393847 10044 ts_tablet_manager.cc:1367] T 518a2226d5754b80996ec134e448c7fb P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:26.394487 10044 raft_consensus.cc:348] T 518a2226d5754b80996ec134e448c7fb P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:26.394663 10044 raft_consensus.cc:374] T 518a2226d5754b80996ec134e448c7fb P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:26.394744 10044 raft_consensus.cc:729] T 518a2226d5754b80996ec134e448c7fb P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:26.394907 10044 consensus_queue.cc:260] T 518a2226d5754b80996ec134e448c7fb P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:26.395269 10044 ts_tablet_manager.cc:1398] T 518a2226d5754b80996ec134e448c7fb P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:26.541213 10004 raft_consensus.cc:482] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:26.541347 10004 raft_consensus.cc:504] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:26.541779 10004 leader_election.cc:288] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:26.542133 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "518a2226d5754b80996ec134e448c7fb" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:26.542265 6791 raft_consensus.cc:2454] T 518a2226d5754b80996ec134e448c7fb P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:26.542548 9146 leader_election.cc:302] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:26.542722 10004 raft_consensus.cc:2781] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:26.542788 10004 raft_consensus.cc:482] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:26.542830 10004 raft_consensus.cc:3037] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:26.543869 10004 raft_consensus.cc:504] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:26.544207 10004 leader_election.cc:288] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:26.544606 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "518a2226d5754b80996ec134e448c7fb" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:26.544703 6791 raft_consensus.cc:3037] T 518a2226d5754b80996ec134e448c7fb P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:26.545699 6791 raft_consensus.cc:2454] T 518a2226d5754b80996ec134e448c7fb P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:26.545876 9342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "518a2226d5754b80996ec134e448c7fb" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:26.545851 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "518a2226d5754b80996ec134e448c7fb" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:26.545960 9146 leader_election.cc:302] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:26.546010 9342 raft_consensus.cc:3037] T 518a2226d5754b80996ec134e448c7fb P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:26.546123 10004 raft_consensus.cc:2781] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:26.546305 10004 raft_consensus.cc:686] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 13:59:26.546455 10004 consensus_queue.cc:237] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:26.547370 9342 raft_consensus.cc:2454] T 518a2226d5754b80996ec134e448c7fb P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:26.548462 6383 catalog_manager.cc:5484] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:26.564498 10027 consensus_queue.cc:1035] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:26.567344 6922 raft_consensus.cc:3037] T d51634c08e2749d4a3bbff4b2a786e48 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:26.569715 9717 consensus_queue.cc:1035] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:26.596740 9937 consensus_queue.cc:1035] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:26.600149 10057 consensus_queue.cc:1035] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:26.643345 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:26.649194 6901 tablet_service.cc:1467] Processing CreateTablet for tablet 7a44f5349cc44be69e9b52f8047d2bb1 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=67fda36210c64eee9922da159b8c91a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:26.649345 9190 tablet_service.cc:1467] Processing CreateTablet for tablet 7a44f5349cc44be69e9b52f8047d2bb1 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=67fda36210c64eee9922da159b8c91a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:26.649451 9323 tablet_service.cc:1467] Processing CreateTablet for tablet 7a44f5349cc44be69e9b52f8047d2bb1 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=67fda36210c64eee9922da159b8c91a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:26.649511 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a44f5349cc44be69e9b52f8047d2bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:26.649623 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a44f5349cc44be69e9b52f8047d2bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:26.649703 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a44f5349cc44be69e9b52f8047d2bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:26.651470 10060 tablet_bootstrap.cc:492] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:26.651695 10043 tablet_bootstrap.cc:492] T 7a44f5349cc44be69e9b52f8047d2bb1 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:26.652132 9323 tablet_service.cc:1467] Processing CreateTablet for tablet 83aef7ca410b4f9c9aa96a432f290ef4 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=67fda36210c64eee9922da159b8c91a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:26.652377 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83aef7ca410b4f9c9aa96a432f290ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:26.652397 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 83aef7ca410b4f9c9aa96a432f290ef4 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=67fda36210c64eee9922da159b8c91a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:26.652554 10043 tablet_bootstrap.cc:654] T 7a44f5349cc44be69e9b52f8047d2bb1 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:26.652673 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83aef7ca410b4f9c9aa96a432f290ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:26.653157 10045 tablet_bootstrap.cc:492] T 7a44f5349cc44be69e9b52f8047d2bb1 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:26.653110 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 83aef7ca410b4f9c9aa96a432f290ef4 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=67fda36210c64eee9922da159b8c91a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:26.653395 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83aef7ca410b4f9c9aa96a432f290ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:26.653548 6508 tablet_service.cc:1467] Processing CreateTablet for tablet bbee7bd521374f99a78d32638260a843 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=67fda36210c64eee9922da159b8c91a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:26.653808 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbee7bd521374f99a78d32638260a843. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:26.654048 10045 tablet_bootstrap.cc:654] T 7a44f5349cc44be69e9b52f8047d2bb1 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:26.654773 9061 tablet_service.cc:1467] Processing CreateTablet for tablet bbee7bd521374f99a78d32638260a843 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=67fda36210c64eee9922da159b8c91a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:26.654979 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbee7bd521374f99a78d32638260a843. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:26.655758 10060 tablet_bootstrap.cc:654] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:26.656392 10063 tablet_bootstrap.cc:492] T 83aef7ca410b4f9c9aa96a432f290ef4 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:26.657058 10064 tablet_bootstrap.cc:492] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:26.657265 10065 tablet_bootstrap.cc:492] T bbee7bd521374f99a78d32638260a843 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:26.657375 10063 tablet_bootstrap.cc:654] T 83aef7ca410b4f9c9aa96a432f290ef4 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:26.657984 10064 tablet_bootstrap.cc:654] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:26.658197 10065 tablet_bootstrap.cc:654] T bbee7bd521374f99a78d32638260a843 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:26.658502 10063 tablet_bootstrap.cc:492] T 83aef7ca410b4f9c9aa96a432f290ef4 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:26.658612 10063 ts_tablet_manager.cc:1367] T 83aef7ca410b4f9c9aa96a432f290ef4 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:26.659161 10064 tablet_bootstrap.cc:492] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:26.659101 10063 raft_consensus.cc:348] T 83aef7ca410b4f9c9aa96a432f290ef4 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:26.659217 10063 raft_consensus.cc:374] T 83aef7ca410b4f9c9aa96a432f290ef4 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:26.659250 10064 ts_tablet_manager.cc:1367] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:26.659416 10043 tablet_bootstrap.cc:492] T 7a44f5349cc44be69e9b52f8047d2bb1 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:26.659250 10063 raft_consensus.cc:729] T 83aef7ca410b4f9c9aa96a432f290ef4 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:26.659502 10043 ts_tablet_manager.cc:1367] T 7a44f5349cc44be69e9b52f8047d2bb1 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 13:59:26.659554 10063 consensus_queue.cc:260] T 83aef7ca410b4f9c9aa96a432f290ef4 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:26.659698 10064 raft_consensus.cc:348] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:26.659806 10064 raft_consensus.cc:374] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:26.659838 10064 raft_consensus.cc:729] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:26.659844 10063 ts_tablet_manager.cc:1398] T 83aef7ca410b4f9c9aa96a432f290ef4 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:26.659953 10064 consensus_queue.cc:260] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:26.659943 10043 raft_consensus.cc:348] T 7a44f5349cc44be69e9b52f8047d2bb1 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:26.660055 10043 raft_consensus.cc:374] T 7a44f5349cc44be69e9b52f8047d2bb1 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:26.660087 10043 raft_consensus.cc:729] T 7a44f5349cc44be69e9b52f8047d2bb1 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:26.660140 9190 tablet_service.cc:1467] Processing CreateTablet for tablet bbee7bd521374f99a78d32638260a843 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=67fda36210c64eee9922da159b8c91a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:26.660188 10043 consensus_queue.cc:260] T 7a44f5349cc44be69e9b52f8047d2bb1 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:26.660214 10064 ts_tablet_manager.cc:1398] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:26.660365 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbee7bd521374f99a78d32638260a843. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:26.660377 10064 tablet_bootstrap.cc:492] T bbee7bd521374f99a78d32638260a843 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:26.660425 10043 ts_tablet_manager.cc:1398] T 7a44f5349cc44be69e9b52f8047d2bb1 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:26.660576 10043 tablet_bootstrap.cc:492] T 83aef7ca410b4f9c9aa96a432f290ef4 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:26.661149 10064 tablet_bootstrap.cc:654] T bbee7bd521374f99a78d32638260a843 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:26.661296 10043 tablet_bootstrap.cc:654] T 83aef7ca410b4f9c9aa96a432f290ef4 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:26.662998 10043 tablet_bootstrap.cc:492] T 83aef7ca410b4f9c9aa96a432f290ef4 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:26.663077 10043 ts_tablet_manager.cc:1367] T 83aef7ca410b4f9c9aa96a432f290ef4 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:59:26.663549 10043 raft_consensus.cc:348] T 83aef7ca410b4f9c9aa96a432f290ef4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:26.663645 10043 raft_consensus.cc:374] T 83aef7ca410b4f9c9aa96a432f290ef4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:26.663693 10043 raft_consensus.cc:729] T 83aef7ca410b4f9c9aa96a432f290ef4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:26.663815 10043 consensus_queue.cc:260] T 83aef7ca410b4f9c9aa96a432f290ef4 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:26.664075 10043 ts_tablet_manager.cc:1398] T 83aef7ca410b4f9c9aa96a432f290ef4 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:26.664371 10045 tablet_bootstrap.cc:492] T 7a44f5349cc44be69e9b52f8047d2bb1 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:26.664440 10045 ts_tablet_manager.cc:1367] T 7a44f5349cc44be69e9b52f8047d2bb1 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 13:59:26.664528 10064 tablet_bootstrap.cc:492] T bbee7bd521374f99a78d32638260a843 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:26.664572 10060 tablet_bootstrap.cc:492] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:26.664605 10064 ts_tablet_manager.cc:1367] T bbee7bd521374f99a78d32638260a843 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 13:59:26.664646 10060 ts_tablet_manager.cc:1367] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20240503 13:59:26.664868 10045 raft_consensus.cc:348] T 7a44f5349cc44be69e9b52f8047d2bb1 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:26.664973 10045 raft_consensus.cc:374] T 7a44f5349cc44be69e9b52f8047d2bb1 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:26.665002 10045 raft_consensus.cc:729] T 7a44f5349cc44be69e9b52f8047d2bb1 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:26.665053 10064 raft_consensus.cc:348] T bbee7bd521374f99a78d32638260a843 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:26.665027 10060 raft_consensus.cc:348] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:26.665153 10060 raft_consensus.cc:374] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:26.665150 10064 raft_consensus.cc:374] T bbee7bd521374f99a78d32638260a843 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:26.665110 10045 consensus_queue.cc:260] T 7a44f5349cc44be69e9b52f8047d2bb1 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:26.665201 10060 raft_consensus.cc:729] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:26.665201 10064 raft_consensus.cc:729] T bbee7bd521374f99a78d32638260a843 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:26.665318 10064 consensus_queue.cc:260] T bbee7bd521374f99a78d32638260a843 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:26.665318 10060 consensus_queue.cc:260] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:26.665432 10045 ts_tablet_manager.cc:1398] T 7a44f5349cc44be69e9b52f8047d2bb1 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:26.665544 10065 tablet_bootstrap.cc:492] T bbee7bd521374f99a78d32638260a843 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:26.665583 10045 tablet_bootstrap.cc:492] T bbee7bd521374f99a78d32638260a843 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:26.665588 10060 ts_tablet_manager.cc:1398] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:26.665620 10065 ts_tablet_manager.cc:1367] T bbee7bd521374f99a78d32638260a843 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:59:26.665705 10064 ts_tablet_manager.cc:1398] T bbee7bd521374f99a78d32638260a843 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:26.666069 10065 raft_consensus.cc:348] T bbee7bd521374f99a78d32638260a843 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:26.666164 10065 raft_consensus.cc:374] T bbee7bd521374f99a78d32638260a843 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:26.666195 10065 raft_consensus.cc:729] T bbee7bd521374f99a78d32638260a843 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:26.666451 10045 tablet_bootstrap.cc:654] T bbee7bd521374f99a78d32638260a843 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:26.666445 10065 consensus_queue.cc:260] T bbee7bd521374f99a78d32638260a843 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:26.666770 10065 ts_tablet_manager.cc:1398] T bbee7bd521374f99a78d32638260a843 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:26.672250 10045 tablet_bootstrap.cc:492] T bbee7bd521374f99a78d32638260a843 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:26.672341 10045 ts_tablet_manager.cc:1367] T bbee7bd521374f99a78d32638260a843 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:59:26.672673 10045 raft_consensus.cc:348] T bbee7bd521374f99a78d32638260a843 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:26.672753 10045 raft_consensus.cc:374] T bbee7bd521374f99a78d32638260a843 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:26.672786 10045 raft_consensus.cc:729] T bbee7bd521374f99a78d32638260a843 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:26.672894 10045 consensus_queue.cc:260] T bbee7bd521374f99a78d32638260a843 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:26.673134 10045 ts_tablet_manager.cc:1398] T bbee7bd521374f99a78d32638260a843 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:26.681761 6383 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:26.685854 9342 raft_consensus.cc:1261] T 518a2226d5754b80996ec134e448c7fb P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:26.686133 6791 raft_consensus.cc:1261] T 518a2226d5754b80996ec134e448c7fb P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:26.686255 10051 consensus_queue.cc:1035] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:26.686506 10004 consensus_queue.cc:1035] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:26.689105 6380 catalog_manager.cc:5576] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931) does not have the latest schema for tablet 518a2226d5754b80996ec134e448c7fb (table rebalancer_dynamic_table [id=5dd58c7354c64075ab0018a952bdc68a]). Expected version 1 got 0
I20240503 13:59:26.690526 10078 tablet.cc:1722] T 518a2226d5754b80996ec134e448c7fb P 4a85e492c8154e9881a9cc2a105d7e64: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:26.690687 6383 catalog_manager.cc:5576] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365) does not have the latest schema for tablet 518a2226d5754b80996ec134e448c7fb (table rebalancer_dynamic_table [id=5dd58c7354c64075ab0018a952bdc68a]). Expected version 1 got 0
I20240503 13:59:26.690953 10078 tablet.cc:1620] T 518a2226d5754b80996ec134e448c7fb P 4a85e492c8154e9881a9cc2a105d7e64: MemRowSet was empty: no flush needed.
I20240503 13:59:26.692134 10029 tablet.cc:1722] T 518a2226d5754b80996ec134e448c7fb P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:26.692380 10029 tablet.cc:1620] T 518a2226d5754b80996ec134e448c7fb P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:26.694191 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=5dd58c7354c64075ab0018a952bdc68a] alter complete (version 1)
I20240503 13:59:26.694461 6383 catalog_manager.cc:5576] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091) does not have the latest schema for tablet 518a2226d5754b80996ec134e448c7fb (table rebalancer_dynamic_table [id=5dd58c7354c64075ab0018a952bdc68a]). Expected version 1 got 0
I20240503 13:59:26.696714 6379 catalog_manager.cc:5576] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091) does not have the latest schema for tablet 518a2226d5754b80996ec134e448c7fb (table rebalancer_dynamic_table [id=5dd58c7354c64075ab0018a952bdc68a]). Expected version 1 got 0
I20240503 13:59:26.695555 10077 tablet.cc:1722] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:26.697268 10077 tablet.cc:1620] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:26.740948 10035 raft_consensus.cc:482] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:26.741055 10035 raft_consensus.cc:504] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:26.741477 10035 leader_election.cc:288] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:26.741782 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83aef7ca410b4f9c9aa96a432f290ef4" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:26.741849 9342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83aef7ca410b4f9c9aa96a432f290ef4" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:26.741910 6660 raft_consensus.cc:2454] T 83aef7ca410b4f9c9aa96a432f290ef4 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:26.741961 9342 raft_consensus.cc:2454] T 83aef7ca410b4f9c9aa96a432f290ef4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:26.742277 9017 leader_election.cc:302] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:26.742476 10035 raft_consensus.cc:2781] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:26.742550 10035 raft_consensus.cc:482] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:26.742590 10035 raft_consensus.cc:3037] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:26.743702 10035 raft_consensus.cc:504] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:26.744009 10035 leader_election.cc:288] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:26.744355 9342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83aef7ca410b4f9c9aa96a432f290ef4" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:26.744482 9342 raft_consensus.cc:3037] T 83aef7ca410b4f9c9aa96a432f290ef4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:26.745221 9342 raft_consensus.cc:2454] T 83aef7ca410b4f9c9aa96a432f290ef4 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:26.745483 9017 leader_election.cc:302] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:26.745620 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83aef7ca410b4f9c9aa96a432f290ef4" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:26.745733 6660 raft_consensus.cc:3037] T 83aef7ca410b4f9c9aa96a432f290ef4 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:26.746639 6660 raft_consensus.cc:2454] T 83aef7ca410b4f9c9aa96a432f290ef4 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:26.746829 10035 raft_consensus.cc:2781] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:26.746913 10035 raft_consensus.cc:686] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 13:59:26.747042 10035 consensus_queue.cc:237] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:26.748942 6383 catalog_manager.cc:5484] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:26.797190 6383 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:26.802505 10077 tablet.cc:1722] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:26.802877 10077 tablet.cc:1620] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:26.802932 10078 tablet.cc:1722] T 518a2226d5754b80996ec134e448c7fb P 4a85e492c8154e9881a9cc2a105d7e64: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:26.803047 10029 tablet.cc:1722] T 518a2226d5754b80996ec134e448c7fb P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:26.803344 10029 tablet.cc:1620] T 518a2226d5754b80996ec134e448c7fb P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:26.804912 10078 tablet.cc:1620] T 518a2226d5754b80996ec134e448c7fb P 4a85e492c8154e9881a9cc2a105d7e64: MemRowSet was empty: no flush needed.
I20240503 13:59:26.807907 6383 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=5dd58c7354c64075ab0018a952bdc68a] alter complete (version 2)
I20240503 13:59:26.809036 10024 raft_consensus.cc:482] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:26.809161 10024 raft_consensus.cc:504] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:26.809648 10024 leader_election.cc:288] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:26.809993 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a44f5349cc44be69e9b52f8047d2bb1" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:26.810117 9211 raft_consensus.cc:2454] T 7a44f5349cc44be69e9b52f8047d2bb1 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:26.810451 6857 leader_election.cc:302] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab1d21935d6049888eb0ceccc3c9d735, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:26.810757 10024 raft_consensus.cc:2781] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:26.810815 10024 raft_consensus.cc:482] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:26.810858 10024 raft_consensus.cc:3037] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:26.811676 10024 raft_consensus.cc:504] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:26.811971 10024 leader_election.cc:288] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:26.812311 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a44f5349cc44be69e9b52f8047d2bb1" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:26.812412 9211 raft_consensus.cc:3037] T 7a44f5349cc44be69e9b52f8047d2bb1 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:26.813076 9211 raft_consensus.cc:2454] T 7a44f5349cc44be69e9b52f8047d2bb1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:26.813331 6857 leader_election.cc:302] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab1d21935d6049888eb0ceccc3c9d735, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:26.813467 10024 raft_consensus.cc:2781] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:26.813645 10024 raft_consensus.cc:686] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:26.813798 10024 consensus_queue.cc:237] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:26.815963 6383 catalog_manager.cc:5484] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:26.817453 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a44f5349cc44be69e9b52f8047d2bb1" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:26.817569 9343 raft_consensus.cc:3037] T 7a44f5349cc44be69e9b52f8047d2bb1 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:26.817734 9342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a44f5349cc44be69e9b52f8047d2bb1" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:26.818406 9343 raft_consensus.cc:2454] T 7a44f5349cc44be69e9b52f8047d2bb1 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:26.901378 10035 raft_consensus.cc:482] T bbee7bd521374f99a78d32638260a843 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:26.901489 10035 raft_consensus.cc:504] T bbee7bd521374f99a78d32638260a843 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:26.901937 10035 leader_election.cc:288] T bbee7bd521374f99a78d32638260a843 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:26.902215 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbee7bd521374f99a78d32638260a843" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:26.902375 6527 raft_consensus.cc:2454] T bbee7bd521374f99a78d32638260a843 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:26.902364 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbee7bd521374f99a78d32638260a843" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:26.902488 9211 raft_consensus.cc:2454] T bbee7bd521374f99a78d32638260a843 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:26.902702 9016 leader_election.cc:302] T bbee7bd521374f99a78d32638260a843 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:26.902873 10035 raft_consensus.cc:2781] T bbee7bd521374f99a78d32638260a843 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:26.902930 10035 raft_consensus.cc:482] T bbee7bd521374f99a78d32638260a843 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:26.902977 10035 raft_consensus.cc:3037] T bbee7bd521374f99a78d32638260a843 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:26.903965 10035 raft_consensus.cc:504] T bbee7bd521374f99a78d32638260a843 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:26.904305 10035 leader_election.cc:288] T bbee7bd521374f99a78d32638260a843 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:26.904578 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbee7bd521374f99a78d32638260a843" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:26.904688 6527 raft_consensus.cc:3037] T bbee7bd521374f99a78d32638260a843 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:26.904677 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbee7bd521374f99a78d32638260a843" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:26.904783 9211 raft_consensus.cc:3037] T bbee7bd521374f99a78d32638260a843 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:26.905663 6527 raft_consensus.cc:2454] T bbee7bd521374f99a78d32638260a843 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:26.905663 9211 raft_consensus.cc:2454] T bbee7bd521374f99a78d32638260a843 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:26.905962 9016 leader_election.cc:302] T bbee7bd521374f99a78d32638260a843 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:26.906122 10035 raft_consensus.cc:2781] T bbee7bd521374f99a78d32638260a843 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:26.906222 10035 raft_consensus.cc:686] T bbee7bd521374f99a78d32638260a843 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 13:59:26.906370 10035 consensus_queue.cc:237] T bbee7bd521374f99a78d32638260a843 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:26.908171 6383 catalog_manager.cc:5484] T bbee7bd521374f99a78d32638260a843 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:26.910143 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:26.914901 10077 tablet.cc:1722] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:26.915469 10078 tablet.cc:1722] T 518a2226d5754b80996ec134e448c7fb P 4a85e492c8154e9881a9cc2a105d7e64: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:26.915490 10029 tablet.cc:1722] T 518a2226d5754b80996ec134e448c7fb P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:26.919992 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=5dd58c7354c64075ab0018a952bdc68a] alter complete (version 3)
I20240503 13:59:27.026793 6380 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:27.026937 6380 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:27.029232 6380 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 518a2226d5754b80996ec134e448c7fb
I20240503 13:59:27.029742 9323 tablet_service.cc:1514] Processing DeleteTablet for tablet 518a2226d5754b80996ec134e448c7fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:27 UTC) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:27.029898 6771 tablet_service.cc:1514] Processing DeleteTablet for tablet 518a2226d5754b80996ec134e448c7fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:27 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:27.029922 9190 tablet_service.cc:1514] Processing DeleteTablet for tablet 518a2226d5754b80996ec134e448c7fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:27 UTC) from {username='slave'} at 127.0.0.1:57960
I20240503 13:59:27.030071 10087 tablet_replica.cc:330] T 518a2226d5754b80996ec134e448c7fb P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:27.030202 10088 tablet_replica.cc:330] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 13:59:27.030190 10089 tablet_replica.cc:330] T 518a2226d5754b80996ec134e448c7fb P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:27.030295 10087 raft_consensus.cc:2229] T 518a2226d5754b80996ec134e448c7fb P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:27.030333 10089 raft_consensus.cc:2229] T 518a2226d5754b80996ec134e448c7fb P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:27.030364 10088 raft_consensus.cc:2229] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:27.030426 10087 raft_consensus.cc:2258] T 518a2226d5754b80996ec134e448c7fb P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:27.030447 10089 raft_consensus.cc:2258] T 518a2226d5754b80996ec134e448c7fb P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:27.030676 10088 raft_consensus.cc:2258] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:27.030993 10087 ts_tablet_manager.cc:1875] T 518a2226d5754b80996ec134e448c7fb P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:27.031186 10089 ts_tablet_manager.cc:1875] T 518a2226d5754b80996ec134e448c7fb P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:27.031231 10088 ts_tablet_manager.cc:1875] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:27.033143 10087 ts_tablet_manager.cc:1888] T 518a2226d5754b80996ec134e448c7fb P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:27.033248 10087 log.cc:1196] T 518a2226d5754b80996ec134e448c7fb P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/518a2226d5754b80996ec134e448c7fb
I20240503 13:59:27.033524 10088 ts_tablet_manager.cc:1888] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:27.033524 10089 ts_tablet_manager.cc:1888] T 518a2226d5754b80996ec134e448c7fb P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:27.033600 10089 log.cc:1196] T 518a2226d5754b80996ec134e448c7fb P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/518a2226d5754b80996ec134e448c7fb
I20240503 13:59:27.033599 10087 ts_tablet_manager.cc:1909] T 518a2226d5754b80996ec134e448c7fb P 0d77752181e944698e809e325e5937e6: Deleting consensus metadata
I20240503 13:59:27.033623 10088 log.cc:1196] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/518a2226d5754b80996ec134e448c7fb
I20240503 13:59:27.033898 10088 ts_tablet_manager.cc:1909] T 518a2226d5754b80996ec134e448c7fb P ab1d21935d6049888eb0ceccc3c9d735: Deleting consensus metadata
I20240503 13:59:27.033931 10089 ts_tablet_manager.cc:1909] T 518a2226d5754b80996ec134e448c7fb P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:27.034417 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet 518a2226d5754b80996ec134e448c7fb (table rebalancer_dynamic_table_new_name [id=5dd58c7354c64075ab0018a952bdc68a]) successfully deleted
I20240503 13:59:27.034659 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet 518a2226d5754b80996ec134e448c7fb (table rebalancer_dynamic_table_new_name [id=5dd58c7354c64075ab0018a952bdc68a]) successfully deleted
I20240503 13:59:27.034893 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet 518a2226d5754b80996ec134e448c7fb (table rebalancer_dynamic_table_new_name [id=5dd58c7354c64075ab0018a952bdc68a]) successfully deleted
I20240503 13:59:27.131163 6380 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:27.137341 9061 tablet_service.cc:1467] Processing CreateTablet for tablet 4448c3a11e0a478fbd19b8d9be2e3dea (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0f8bdea03abb458bb22603a41fa077a7]), partition=
I20240503 13:59:27.137459 9190 tablet_service.cc:1467] Processing CreateTablet for tablet 4448c3a11e0a478fbd19b8d9be2e3dea (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0f8bdea03abb458bb22603a41fa077a7]), partition=
I20240503 13:59:27.137459 8927 tablet_service.cc:1467] Processing CreateTablet for tablet 4448c3a11e0a478fbd19b8d9be2e3dea (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0f8bdea03abb458bb22603a41fa077a7]), partition=
I20240503 13:59:27.137616 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4448c3a11e0a478fbd19b8d9be2e3dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:27.137691 8927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4448c3a11e0a478fbd19b8d9be2e3dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:27.137691 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4448c3a11e0a478fbd19b8d9be2e3dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:27.139323 10064 tablet_bootstrap.cc:492] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:27.139753 10045 tablet_bootstrap.cc:492] T 4448c3a11e0a478fbd19b8d9be2e3dea P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:27.140275 10064 tablet_bootstrap.cc:654] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:27.140645 10045 tablet_bootstrap.cc:654] T 4448c3a11e0a478fbd19b8d9be2e3dea P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:27.141486 10092 tablet_bootstrap.cc:492] T 4448c3a11e0a478fbd19b8d9be2e3dea P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:27.141636 10064 tablet_bootstrap.cc:492] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:27.141722 10064 ts_tablet_manager.cc:1367] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:27.141829 10045 tablet_bootstrap.cc:492] T 4448c3a11e0a478fbd19b8d9be2e3dea P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:27.141912 10045 ts_tablet_manager.cc:1367] T 4448c3a11e0a478fbd19b8d9be2e3dea P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:27.142421 10092 tablet_bootstrap.cc:654] T 4448c3a11e0a478fbd19b8d9be2e3dea P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:27.142379 10064 raft_consensus.cc:348] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:27.142480 10064 raft_consensus.cc:374] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:27.142511 10064 raft_consensus.cc:729] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:27.142462 10045 raft_consensus.cc:348] T 4448c3a11e0a478fbd19b8d9be2e3dea P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:27.142580 10045 raft_consensus.cc:374] T 4448c3a11e0a478fbd19b8d9be2e3dea P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:27.142616 10045 raft_consensus.cc:729] T 4448c3a11e0a478fbd19b8d9be2e3dea P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:27.142751 10045 consensus_queue.cc:260] T 4448c3a11e0a478fbd19b8d9be2e3dea P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:27.143123 10045 ts_tablet_manager.cc:1398] T 4448c3a11e0a478fbd19b8d9be2e3dea P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:27.143368 10064 consensus_queue.cc:260] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:27.143649 10064 ts_tablet_manager.cc:1398] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:27.145553 10092 tablet_bootstrap.cc:492] T 4448c3a11e0a478fbd19b8d9be2e3dea P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:27.145686 10092 ts_tablet_manager.cc:1367] T 4448c3a11e0a478fbd19b8d9be2e3dea P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:27.146134 10092 raft_consensus.cc:348] T 4448c3a11e0a478fbd19b8d9be2e3dea P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:27.146291 10092 raft_consensus.cc:374] T 4448c3a11e0a478fbd19b8d9be2e3dea P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:27.146348 10092 raft_consensus.cc:729] T 4448c3a11e0a478fbd19b8d9be2e3dea P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:27.146517 10092 consensus_queue.cc:260] T 4448c3a11e0a478fbd19b8d9be2e3dea P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:27.146812 10092 ts_tablet_manager.cc:1398] T 4448c3a11e0a478fbd19b8d9be2e3dea P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:27.248672 10083 consensus_queue.cc:1035] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:27.253515 10098 consensus_queue.cc:1035] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:27.270185 10035 consensus_queue.cc:1035] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:27.274075 9987 consensus_queue.cc:1035] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:27.320283 10035 consensus_queue.cc:1035] T bbee7bd521374f99a78d32638260a843 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:27.323968 10105 consensus_queue.cc:1035] T bbee7bd521374f99a78d32638260a843 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:27.406241 10105 raft_consensus.cc:482] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:27.406399 10105 raft_consensus.cc:504] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:27.406821 10105 leader_election.cc:288] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:27.407136 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4448c3a11e0a478fbd19b8d9be2e3dea" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:27.407222 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4448c3a11e0a478fbd19b8d9be2e3dea" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:27.407310 9211 raft_consensus.cc:2454] T 4448c3a11e0a478fbd19b8d9be2e3dea P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:27.407411 8947 raft_consensus.cc:2454] T 4448c3a11e0a478fbd19b8d9be2e3dea P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:27.407625 9016 leader_election.cc:302] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:27.407909 10105 raft_consensus.cc:2781] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:27.407968 10105 raft_consensus.cc:482] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:27.408073 10105 raft_consensus.cc:3037] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:27.409113 10105 raft_consensus.cc:504] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:27.409427 10105 leader_election.cc:288] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:27.409780 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4448c3a11e0a478fbd19b8d9be2e3dea" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:27.409888 8947 raft_consensus.cc:3037] T 4448c3a11e0a478fbd19b8d9be2e3dea P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:27.410919 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4448c3a11e0a478fbd19b8d9be2e3dea" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:27.411048 9205 raft_consensus.cc:3037] T 4448c3a11e0a478fbd19b8d9be2e3dea P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:27.411047 8947 raft_consensus.cc:2454] T 4448c3a11e0a478fbd19b8d9be2e3dea P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:27.411363 9016 leader_election.cc:302] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:27.411509 10105 raft_consensus.cc:2781] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:27.411612 10105 raft_consensus.cc:686] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 13:59:27.411734 10105 consensus_queue.cc:237] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:27.412112 9205 raft_consensus.cc:2454] T 4448c3a11e0a478fbd19b8d9be2e3dea P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:27.413579 6379 catalog_manager.cc:5484] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:27.458210 6379 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:27.464159 9323 tablet_service.cc:1467] Processing CreateTablet for tablet 64135eb6ef574c95b3677e32d66748a4 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=c8e9e03cbc904134b1bfd2851cf3b5a7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:27.464313 6771 tablet_service.cc:1467] Processing CreateTablet for tablet 64135eb6ef574c95b3677e32d66748a4 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=c8e9e03cbc904134b1bfd2851cf3b5a7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:27.464502 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64135eb6ef574c95b3677e32d66748a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:27.464612 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64135eb6ef574c95b3677e32d66748a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:27.465119 6769 tablet_service.cc:1467] Processing CreateTablet for tablet 957b89de328d434d8ea03a59bc93fdcb (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=c8e9e03cbc904134b1bfd2851cf3b5a7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:27.465204 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 42061f6b71d74b98814092ba97a75086 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=c8e9e03cbc904134b1bfd2851cf3b5a7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:27.465368 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 957b89de328d434d8ea03a59bc93fdcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:27.465492 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42061f6b71d74b98814092ba97a75086. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:27.467084 9061 tablet_service.cc:1467] Processing CreateTablet for tablet 64135eb6ef574c95b3677e32d66748a4 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=c8e9e03cbc904134b1bfd2851cf3b5a7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:27.467396 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64135eb6ef574c95b3677e32d66748a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:27.468262 9190 tablet_service.cc:1467] Processing CreateTablet for tablet 957b89de328d434d8ea03a59bc93fdcb (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=c8e9e03cbc904134b1bfd2851cf3b5a7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:27.468418 10111 tablet_bootstrap.cc:492] T 42061f6b71d74b98814092ba97a75086 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:27.468523 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 957b89de328d434d8ea03a59bc93fdcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:27.468778 10109 tablet_bootstrap.cc:492] T 64135eb6ef574c95b3677e32d66748a4 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:27.469424 10111 tablet_bootstrap.cc:654] T 42061f6b71d74b98814092ba97a75086 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:27.469699 10109 tablet_bootstrap.cc:654] T 64135eb6ef574c95b3677e32d66748a4 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:27.469856 9191 tablet_service.cc:1467] Processing CreateTablet for tablet 42061f6b71d74b98814092ba97a75086 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=c8e9e03cbc904134b1bfd2851cf3b5a7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:27.470058 9191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42061f6b71d74b98814092ba97a75086. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:27.470769 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 42061f6b71d74b98814092ba97a75086 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=c8e9e03cbc904134b1bfd2851cf3b5a7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:27.470988 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42061f6b71d74b98814092ba97a75086. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:27.471079 10111 tablet_bootstrap.cc:492] T 42061f6b71d74b98814092ba97a75086 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:27.471139 10111 ts_tablet_manager.cc:1367] T 42061f6b71d74b98814092ba97a75086 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:27.471565 10111 raft_consensus.cc:348] T 42061f6b71d74b98814092ba97a75086 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:27.471669 10111 raft_consensus.cc:374] T 42061f6b71d74b98814092ba97a75086 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:27.471621 8927 tablet_service.cc:1467] Processing CreateTablet for tablet 957b89de328d434d8ea03a59bc93fdcb (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=c8e9e03cbc904134b1bfd2851cf3b5a7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:27.471698 10111 raft_consensus.cc:729] T 42061f6b71d74b98814092ba97a75086 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:27.471843 10111 consensus_queue.cc:260] T 42061f6b71d74b98814092ba97a75086 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:27.471905 8927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 957b89de328d434d8ea03a59bc93fdcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:27.471923 10045 tablet_bootstrap.cc:492] T 957b89de328d434d8ea03a59bc93fdcb P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:27.472115 10111 ts_tablet_manager.cc:1398] T 42061f6b71d74b98814092ba97a75086 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:27.472805 10045 tablet_bootstrap.cc:654] T 957b89de328d434d8ea03a59bc93fdcb P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:27.474056 10064 tablet_bootstrap.cc:492] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:27.474438 10092 tablet_bootstrap.cc:492] T 957b89de328d434d8ea03a59bc93fdcb P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:27.474814 10064 tablet_bootstrap.cc:654] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:27.475136 10092 tablet_bootstrap.cc:654] T 957b89de328d434d8ea03a59bc93fdcb P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:27.475872 10064 tablet_bootstrap.cc:492] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:27.475952 10064 ts_tablet_manager.cc:1367] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 13:59:27.476431 10064 raft_consensus.cc:348] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:27.476552 10064 raft_consensus.cc:374] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:27.476583 10064 raft_consensus.cc:729] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:27.476696 10064 consensus_queue.cc:260] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:27.476985 10064 ts_tablet_manager.cc:1398] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:27.477098 10045 tablet_bootstrap.cc:492] T 957b89de328d434d8ea03a59bc93fdcb P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:27.477149 10064 tablet_bootstrap.cc:492] T 42061f6b71d74b98814092ba97a75086 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:27.477173 10045 ts_tablet_manager.cc:1367] T 957b89de328d434d8ea03a59bc93fdcb P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:27.477617 10045 raft_consensus.cc:348] T 957b89de328d434d8ea03a59bc93fdcb P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:27.477722 10045 raft_consensus.cc:374] T 957b89de328d434d8ea03a59bc93fdcb P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:27.477753 10045 raft_consensus.cc:729] T 957b89de328d434d8ea03a59bc93fdcb P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:27.477860 10045 consensus_queue.cc:260] T 957b89de328d434d8ea03a59bc93fdcb P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:27.478145 10045 ts_tablet_manager.cc:1398] T 957b89de328d434d8ea03a59bc93fdcb P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:27.478317 10045 tablet_bootstrap.cc:492] T 42061f6b71d74b98814092ba97a75086 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:27.479780 10064 tablet_bootstrap.cc:654] T 42061f6b71d74b98814092ba97a75086 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:27.480815 10045 tablet_bootstrap.cc:654] T 42061f6b71d74b98814092ba97a75086 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:27.481526 10110 tablet_bootstrap.cc:492] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:27.481730 10064 tablet_bootstrap.cc:492] T 42061f6b71d74b98814092ba97a75086 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:27.482074 10064 ts_tablet_manager.cc:1367] T 42061f6b71d74b98814092ba97a75086 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240503 13:59:27.482568 10064 raft_consensus.cc:348] T 42061f6b71d74b98814092ba97a75086 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:27.482718 10064 raft_consensus.cc:374] T 42061f6b71d74b98814092ba97a75086 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:27.482731 10110 tablet_bootstrap.cc:654] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:27.482801 10064 raft_consensus.cc:729] T 42061f6b71d74b98814092ba97a75086 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:27.482939 10064 consensus_queue.cc:260] T 42061f6b71d74b98814092ba97a75086 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:27.483254 10064 ts_tablet_manager.cc:1398] T 42061f6b71d74b98814092ba97a75086 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:27.483804 10092 tablet_bootstrap.cc:492] T 957b89de328d434d8ea03a59bc93fdcb P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:27.483888 10092 ts_tablet_manager.cc:1367] T 957b89de328d434d8ea03a59bc93fdcb P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240503 13:59:27.484375 10092 raft_consensus.cc:348] T 957b89de328d434d8ea03a59bc93fdcb P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:27.484483 10092 raft_consensus.cc:374] T 957b89de328d434d8ea03a59bc93fdcb P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:27.484515 10092 raft_consensus.cc:729] T 957b89de328d434d8ea03a59bc93fdcb P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:27.484645 10092 consensus_queue.cc:260] T 957b89de328d434d8ea03a59bc93fdcb P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:27.484902 10092 ts_tablet_manager.cc:1398] T 957b89de328d434d8ea03a59bc93fdcb P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:27.486052 10045 tablet_bootstrap.cc:492] T 42061f6b71d74b98814092ba97a75086 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:27.486151 10045 ts_tablet_manager.cc:1367] T 42061f6b71d74b98814092ba97a75086 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:59:27.486563 10045 raft_consensus.cc:348] T 42061f6b71d74b98814092ba97a75086 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:27.486660 10045 raft_consensus.cc:374] T 42061f6b71d74b98814092ba97a75086 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:27.486689 10045 raft_consensus.cc:729] T 42061f6b71d74b98814092ba97a75086 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:27.486866 10045 consensus_queue.cc:260] T 42061f6b71d74b98814092ba97a75086 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:27.487109 10045 ts_tablet_manager.cc:1398] T 42061f6b71d74b98814092ba97a75086 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:27.487582 10110 tablet_bootstrap.cc:492] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:27.487665 10110 ts_tablet_manager.cc:1367] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 13:59:27.487884 10109 tablet_bootstrap.cc:492] T 64135eb6ef574c95b3677e32d66748a4 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:27.487970 10109 ts_tablet_manager.cc:1367] T 64135eb6ef574c95b3677e32d66748a4 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 13:59:27.488335 10109 raft_consensus.cc:348] T 64135eb6ef574c95b3677e32d66748a4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:27.488433 10109 raft_consensus.cc:374] T 64135eb6ef574c95b3677e32d66748a4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:27.488462 10109 raft_consensus.cc:729] T 64135eb6ef574c95b3677e32d66748a4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:27.488574 10109 consensus_queue.cc:260] T 64135eb6ef574c95b3677e32d66748a4 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:27.488849 10109 ts_tablet_manager.cc:1398] T 64135eb6ef574c95b3677e32d66748a4 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:27.489173 10110 raft_consensus.cc:348] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:27.489372 10110 raft_consensus.cc:374] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:27.489468 10110 raft_consensus.cc:729] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:27.489602 10110 consensus_queue.cc:260] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:27.489969 10110 ts_tablet_manager.cc:1398] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:27.490295 10110 tablet_bootstrap.cc:492] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:27.491181 10110 tablet_bootstrap.cc:654] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:27.492187 10110 tablet_bootstrap.cc:492] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:27.492270 10110 ts_tablet_manager.cc:1367] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 13:59:27.492595 10110 raft_consensus.cc:348] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:27.492659 10110 raft_consensus.cc:374] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:27.492677 10110 raft_consensus.cc:729] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:27.492777 10110 consensus_queue.cc:260] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:27.493062 10110 ts_tablet_manager.cc:1398] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:27.506392 10073 raft_consensus.cc:482] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:27.506500 10073 raft_consensus.cc:504] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:27.506889 10073 leader_election.cc:288] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:27.507180 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64135eb6ef574c95b3677e32d66748a4" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:27.507233 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64135eb6ef574c95b3677e32d66748a4" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:27.507357 9343 raft_consensus.cc:2454] T 64135eb6ef574c95b3677e32d66748a4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:27.507396 9081 raft_consensus.cc:2454] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:27.507619 6727 leader_election.cc:302] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:27.507761 10073 raft_consensus.cc:2781] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:27.507849 10073 raft_consensus.cc:482] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:27.507895 10073 raft_consensus.cc:3037] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:27.508785 10073 raft_consensus.cc:504] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:27.509106 10073 leader_election.cc:288] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:27.509421 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64135eb6ef574c95b3677e32d66748a4" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:27.509459 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64135eb6ef574c95b3677e32d66748a4" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:27.509518 9343 raft_consensus.cc:3037] T 64135eb6ef574c95b3677e32d66748a4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:27.509562 9081 raft_consensus.cc:3037] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:27.510246 9081 raft_consensus.cc:2454] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:27.510253 9343 raft_consensus.cc:2454] T 64135eb6ef574c95b3677e32d66748a4 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:27.510555 6726 leader_election.cc:302] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:27.510779 10073 raft_consensus.cc:2781] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:27.510869 10073 raft_consensus.cc:686] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:27.511005 10073 consensus_queue.cc:237] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:27.512982 6379 catalog_manager.cc:5484] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:27.531644 10073 raft_consensus.cc:482] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:27.531745 10073 raft_consensus.cc:504] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:27.532116 10073 leader_election.cc:288] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:27.532480 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "957b89de328d434d8ea03a59bc93fdcb" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:27.532524 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "957b89de328d434d8ea03a59bc93fdcb" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:27.532615 8947 raft_consensus.cc:2454] T 957b89de328d434d8ea03a59bc93fdcb P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:27.532673 9205 raft_consensus.cc:2454] T 957b89de328d434d8ea03a59bc93fdcb P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:27.533041 6726 leader_election.cc:302] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:27.533133 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:27.533218 10073 raft_consensus.cc:2781] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:27.533270 10073 raft_consensus.cc:482] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:27.533315 10073 raft_consensus.cc:3037] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:27.534109 10073 raft_consensus.cc:504] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:27.534452 10073 leader_election.cc:288] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:27.534798 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "957b89de328d434d8ea03a59bc93fdcb" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:27.534924 9205 raft_consensus.cc:3037] T 957b89de328d434d8ea03a59bc93fdcb P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:27.534989 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "957b89de328d434d8ea03a59bc93fdcb" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:27.535089 8947 raft_consensus.cc:3037] T 957b89de328d434d8ea03a59bc93fdcb P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:27.535907 9205 raft_consensus.cc:2454] T 957b89de328d434d8ea03a59bc93fdcb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:27.535991 8947 raft_consensus.cc:2454] T 957b89de328d434d8ea03a59bc93fdcb P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:27.536202 6726 leader_election.cc:302] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:27.536998 10073 raft_consensus.cc:2781] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:27.537142 10073 raft_consensus.cc:686] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:27.537292 10073 consensus_queue.cc:237] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:27.538110 9205 raft_consensus.cc:1261] T 4448c3a11e0a478fbd19b8d9be2e3dea P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:27.538480 10035 consensus_queue.cc:1035] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:27.538609 8947 raft_consensus.cc:1261] T 4448c3a11e0a478fbd19b8d9be2e3dea P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:27.538973 10035 consensus_queue.cc:1035] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:27.540657 6383 catalog_manager.cc:5576] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091) does not have the latest schema for tablet 4448c3a11e0a478fbd19b8d9be2e3dea (table rebalancer_dynamic_table [id=0f8bdea03abb458bb22603a41fa077a7]). Expected version 1 got 0
I20240503 13:59:27.540935 6379 catalog_manager.cc:5484] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:27.541886 6383 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet 4448c3a11e0a478fbd19b8d9be2e3dea (table rebalancer_dynamic_table [id=0f8bdea03abb458bb22603a41fa077a7]). Expected version 1 got 0
I20240503 13:59:27.543177 6383 catalog_manager.cc:5576] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481) does not have the latest schema for tablet 4448c3a11e0a478fbd19b8d9be2e3dea (table rebalancer_dynamic_table [id=0f8bdea03abb458bb22603a41fa077a7]). Expected version 1 got 0
I20240503 13:59:27.543200 10127 tablet.cc:1722] T 4448c3a11e0a478fbd19b8d9be2e3dea P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:27.543493 10127 tablet.cc:1620] T 4448c3a11e0a478fbd19b8d9be2e3dea P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:27.543694 6379 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet 4448c3a11e0a478fbd19b8d9be2e3dea (table rebalancer_dynamic_table [id=0f8bdea03abb458bb22603a41fa077a7]). Expected version 1 got 0
I20240503 13:59:27.543800 10126 tablet.cc:1722] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:27.544651 10128 tablet.cc:1722] T 4448c3a11e0a478fbd19b8d9be2e3dea P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:27.545785 10128 tablet.cc:1620] T 4448c3a11e0a478fbd19b8d9be2e3dea P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:27.546036 6383 catalog_manager.cc:5576] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091) does not have the latest schema for tablet 4448c3a11e0a478fbd19b8d9be2e3dea (table rebalancer_dynamic_table [id=0f8bdea03abb458bb22603a41fa077a7]). Expected version 1 got 0
I20240503 13:59:27.546618 10126 tablet.cc:1620] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:27.547694 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=0f8bdea03abb458bb22603a41fa077a7] alter complete (version 1)
I20240503 13:59:27.650580 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:27.655369 10126 tablet.cc:1722] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:27.655699 10126 tablet.cc:1620] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:27.655773 10127 tablet.cc:1722] T 4448c3a11e0a478fbd19b8d9be2e3dea P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:27.656091 10127 tablet.cc:1620] T 4448c3a11e0a478fbd19b8d9be2e3dea P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:27.656252 10128 tablet.cc:1722] T 4448c3a11e0a478fbd19b8d9be2e3dea P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:27.656582 10128 tablet.cc:1620] T 4448c3a11e0a478fbd19b8d9be2e3dea P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:27.660495 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=0f8bdea03abb458bb22603a41fa077a7] alter complete (version 2)
I20240503 13:59:27.739702 10081 raft_consensus.cc:482] T 42061f6b71d74b98814092ba97a75086 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:27.739821 10081 raft_consensus.cc:504] T 42061f6b71d74b98814092ba97a75086 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:27.740272 10081 leader_election.cc:288] T 42061f6b71d74b98814092ba97a75086 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:27.740880 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42061f6b71d74b98814092ba97a75086" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:27.741019 9081 raft_consensus.cc:2454] T 42061f6b71d74b98814092ba97a75086 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:27.741310 9147 leader_election.cc:302] T 42061f6b71d74b98814092ba97a75086 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:27.741464 10081 raft_consensus.cc:2781] T 42061f6b71d74b98814092ba97a75086 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:27.741513 10081 raft_consensus.cc:482] T 42061f6b71d74b98814092ba97a75086 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:27.741559 10081 raft_consensus.cc:3037] T 42061f6b71d74b98814092ba97a75086 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:27.742619 10081 raft_consensus.cc:504] T 42061f6b71d74b98814092ba97a75086 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:27.742955 10081 leader_election.cc:288] T 42061f6b71d74b98814092ba97a75086 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Requested vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:27.743749 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42061f6b71d74b98814092ba97a75086" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:27.743861 9081 raft_consensus.cc:3037] T 42061f6b71d74b98814092ba97a75086 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:27.744716 9081 raft_consensus.cc:2454] T 42061f6b71d74b98814092ba97a75086 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:27.745019 9147 leader_election.cc:302] T 42061f6b71d74b98814092ba97a75086 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:27.745193 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42061f6b71d74b98814092ba97a75086" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:27.745329 6659 raft_consensus.cc:2454] T 42061f6b71d74b98814092ba97a75086 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:27.745307 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42061f6b71d74b98814092ba97a75086" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:27.745417 6658 raft_consensus.cc:3037] T 42061f6b71d74b98814092ba97a75086 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:27.745580 10081 raft_consensus.cc:2781] T 42061f6b71d74b98814092ba97a75086 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:27.745672 10081 raft_consensus.cc:686] T 42061f6b71d74b98814092ba97a75086 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 13:59:27.745800 10081 consensus_queue.cc:237] T 42061f6b71d74b98814092ba97a75086 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:27.746338 6658 raft_consensus.cc:2454] T 42061f6b71d74b98814092ba97a75086 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:27.747648 6379 catalog_manager.cc:5484] T 42061f6b71d74b98814092ba97a75086 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:27.762962 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:27.767549 10126 tablet.cc:1722] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:27.768623 10127 tablet.cc:1722] T 4448c3a11e0a478fbd19b8d9be2e3dea P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:27.769317 10128 tablet.cc:1722] T 4448c3a11e0a478fbd19b8d9be2e3dea P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:27.772622 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=0f8bdea03abb458bb22603a41fa077a7] alter complete (version 3)
I20240503 13:59:27.874902 6383 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:27.875085 6383 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:27.877017 6383 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 4448c3a11e0a478fbd19b8d9be2e3dea
I20240503 13:59:27.877583 9058 tablet_service.cc:1514] Processing DeleteTablet for tablet 4448c3a11e0a478fbd19b8d9be2e3dea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:27 UTC) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:27.877617 9191 tablet_service.cc:1514] Processing DeleteTablet for tablet 4448c3a11e0a478fbd19b8d9be2e3dea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:27 UTC) from {username='slave'} at 127.0.0.1:57960
I20240503 13:59:27.877662 8927 tablet_service.cc:1514] Processing DeleteTablet for tablet 4448c3a11e0a478fbd19b8d9be2e3dea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:27 UTC) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:27.877923 10133 tablet_replica.cc:330] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:27.877995 10135 tablet_replica.cc:330] T 4448c3a11e0a478fbd19b8d9be2e3dea P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:27.878055 10134 tablet_replica.cc:330] T 4448c3a11e0a478fbd19b8d9be2e3dea P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 13:59:27.878080 10133 raft_consensus.cc:2229] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:27.878154 10135 raft_consensus.cc:2229] T 4448c3a11e0a478fbd19b8d9be2e3dea P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:27.878201 10134 raft_consensus.cc:2229] T 4448c3a11e0a478fbd19b8d9be2e3dea P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:27.878275 10135 raft_consensus.cc:2258] T 4448c3a11e0a478fbd19b8d9be2e3dea P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:27.878332 10134 raft_consensus.cc:2258] T 4448c3a11e0a478fbd19b8d9be2e3dea P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:27.878350 10133 raft_consensus.cc:2258] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:27.878916 10135 ts_tablet_manager.cc:1875] T 4448c3a11e0a478fbd19b8d9be2e3dea P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:27.879011 10134 ts_tablet_manager.cc:1875] T 4448c3a11e0a478fbd19b8d9be2e3dea P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:27.881328 10134 ts_tablet_manager.cc:1888] T 4448c3a11e0a478fbd19b8d9be2e3dea P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:27.881363 10133 ts_tablet_manager.cc:1875] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:27.881469 10134 log.cc:1196] T 4448c3a11e0a478fbd19b8d9be2e3dea P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/4448c3a11e0a478fbd19b8d9be2e3dea
I20240503 13:59:27.881778 10134 ts_tablet_manager.cc:1909] T 4448c3a11e0a478fbd19b8d9be2e3dea P ab1d21935d6049888eb0ceccc3c9d735: Deleting consensus metadata
I20240503 13:59:27.882632 10135 ts_tablet_manager.cc:1888] T 4448c3a11e0a478fbd19b8d9be2e3dea P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:27.882629 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet 4448c3a11e0a478fbd19b8d9be2e3dea (table rebalancer_dynamic_table_new_name [id=0f8bdea03abb458bb22603a41fa077a7]) successfully deleted
I20240503 13:59:27.882710 10135 log.cc:1196] T 4448c3a11e0a478fbd19b8d9be2e3dea P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/4448c3a11e0a478fbd19b8d9be2e3dea
I20240503 13:59:27.882977 10135 ts_tablet_manager.cc:1909] T 4448c3a11e0a478fbd19b8d9be2e3dea P 06ca8dbc004c42b99faa527a4e6f9545: Deleting consensus metadata
I20240503 13:59:27.883481 10133 ts_tablet_manager.cc:1888] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:27.883555 10133 log.cc:1196] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/4448c3a11e0a478fbd19b8d9be2e3dea
I20240503 13:59:27.883605 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 4448c3a11e0a478fbd19b8d9be2e3dea (table rebalancer_dynamic_table_new_name [id=0f8bdea03abb458bb22603a41fa077a7]) successfully deleted
I20240503 13:59:27.883805 10133 ts_tablet_manager.cc:1909] T 4448c3a11e0a478fbd19b8d9be2e3dea P 8d574fb4a89d4eea947c859470d44dd4: Deleting consensus metadata
I20240503 13:59:27.884337 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet 4448c3a11e0a478fbd19b8d9be2e3dea (table rebalancer_dynamic_table_new_name [id=0f8bdea03abb458bb22603a41fa077a7]) successfully deleted
I20240503 13:59:27.965610 10073 consensus_queue.cc:1035] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:27.969969 10139 consensus_queue.cc:1035] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:27.978829 6379 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:27.984774 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc91db6e866464f89d3e797e976f020 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4f197d059f224052b2ca8471a151015e]), partition=
I20240503 13:59:27.985005 6901 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc91db6e866464f89d3e797e976f020 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4f197d059f224052b2ca8471a151015e]), partition=
I20240503 13:59:27.985005 8927 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc91db6e866464f89d3e797e976f020 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4f197d059f224052b2ca8471a151015e]), partition=
I20240503 13:59:27.985070 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc91db6e866464f89d3e797e976f020. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:27.985230 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc91db6e866464f89d3e797e976f020. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:27.985230 8927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc91db6e866464f89d3e797e976f020. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:27.986837 10142 tablet_bootstrap.cc:492] T 3bc91db6e866464f89d3e797e976f020 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:27.987643 10144 tablet_bootstrap.cc:492] T 3bc91db6e866464f89d3e797e976f020 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:27.987682 10143 tablet_bootstrap.cc:492] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:27.987813 10142 tablet_bootstrap.cc:654] T 3bc91db6e866464f89d3e797e976f020 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:27.988433 10144 tablet_bootstrap.cc:654] T 3bc91db6e866464f89d3e797e976f020 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:27.988747 10143 tablet_bootstrap.cc:654] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:27.989820 10144 tablet_bootstrap.cc:492] T 3bc91db6e866464f89d3e797e976f020 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:27.989899 10142 tablet_bootstrap.cc:492] T 3bc91db6e866464f89d3e797e976f020 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:27.989907 10144 ts_tablet_manager.cc:1367] T 3bc91db6e866464f89d3e797e976f020 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:27.990015 10142 ts_tablet_manager.cc:1367] T 3bc91db6e866464f89d3e797e976f020 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:27.990288 10144 raft_consensus.cc:348] T 3bc91db6e866464f89d3e797e976f020 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:27.990383 10144 raft_consensus.cc:374] T 3bc91db6e866464f89d3e797e976f020 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:27.990404 10144 raft_consensus.cc:729] T 3bc91db6e866464f89d3e797e976f020 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:27.990427 10142 raft_consensus.cc:348] T 3bc91db6e866464f89d3e797e976f020 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:27.990490 10144 consensus_queue.cc:260] T 3bc91db6e866464f89d3e797e976f020 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:27.990545 10142 raft_consensus.cc:374] T 3bc91db6e866464f89d3e797e976f020 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:27.990581 10142 raft_consensus.cc:729] T 3bc91db6e866464f89d3e797e976f020 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:27.990749 10144 ts_tablet_manager.cc:1398] T 3bc91db6e866464f89d3e797e976f020 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:27.990794 10142 consensus_queue.cc:260] T 3bc91db6e866464f89d3e797e976f020 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:27.991158 10142 ts_tablet_manager.cc:1398] T 3bc91db6e866464f89d3e797e976f020 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:27.991701 10143 tablet_bootstrap.cc:492] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:27.991786 10143 ts_tablet_manager.cc:1367] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:27.992237 10143 raft_consensus.cc:348] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:27.992341 10143 raft_consensus.cc:374] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:27.992373 10143 raft_consensus.cc:729] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:27.992489 10143 consensus_queue.cc:260] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:27.992777 10143 ts_tablet_manager.cc:1398] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:28.081570 10091 raft_consensus.cc:482] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:28.081718 10091 raft_consensus.cc:504] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:28.082084 10091 leader_election.cc:288] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:28.082476 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc91db6e866464f89d3e797e976f020" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:28.082509 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc91db6e866464f89d3e797e976f020" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:28.082597 9081 raft_consensus.cc:2454] T 3bc91db6e866464f89d3e797e976f020 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:28.082724 6920 raft_consensus.cc:2454] T 3bc91db6e866464f89d3e797e976f020 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:28.082908 8883 leader_election.cc:302] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:28.083096 10091 raft_consensus.cc:2781] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:28.083156 10091 raft_consensus.cc:482] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:28.083209 10091 raft_consensus.cc:3037] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:28.084087 10091 raft_consensus.cc:504] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:28.084375 10091 leader_election.cc:288] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:28.084695 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc91db6e866464f89d3e797e976f020" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:28.084762 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc91db6e866464f89d3e797e976f020" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:28.084803 9081 raft_consensus.cc:3037] T 3bc91db6e866464f89d3e797e976f020 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:28.084893 6920 raft_consensus.cc:3037] T 3bc91db6e866464f89d3e797e976f020 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:28.085634 9081 raft_consensus.cc:2454] T 3bc91db6e866464f89d3e797e976f020 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:28.085836 6920 raft_consensus.cc:2454] T 3bc91db6e866464f89d3e797e976f020 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:28.085915 8883 leader_election.cc:302] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:28.086076 10091 raft_consensus.cc:2781] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:28.086266 10091 raft_consensus.cc:686] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 13:59:28.086416 10091 consensus_queue.cc:237] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:28.088136 6380 catalog_manager.cc:5484] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:28.110880 10139 consensus_queue.cc:1035] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:28.114744 10139 consensus_queue.cc:1035] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:28.191319 6383 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:28.195261 6920 raft_consensus.cc:1261] T 3bc91db6e866464f89d3e797e976f020 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:28.195415 9081 raft_consensus.cc:1261] T 3bc91db6e866464f89d3e797e976f020 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:28.195735 10091 consensus_queue.cc:1035] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:28.195887 10148 consensus_queue.cc:1035] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:28.198244 10127 tablet.cc:1722] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:28.198406 6383 catalog_manager.cc:5576] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481) does not have the latest schema for tablet 3bc91db6e866464f89d3e797e976f020 (table rebalancer_dynamic_table [id=4f197d059f224052b2ca8471a151015e]). Expected version 1 got 0
I20240503 13:59:28.198597 10127 tablet.cc:1620] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:28.199049 10126 tablet.cc:1722] T 3bc91db6e866464f89d3e797e976f020 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:28.199344 10126 tablet.cc:1620] T 3bc91db6e866464f89d3e797e976f020 P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:28.200042 6383 catalog_manager.cc:5576] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531) does not have the latest schema for tablet 3bc91db6e866464f89d3e797e976f020 (table rebalancer_dynamic_table [id=4f197d059f224052b2ca8471a151015e]). Expected version 1 got 0
I20240503 13:59:28.203990 6383 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=4f197d059f224052b2ca8471a151015e] alter complete (version 1)
I20240503 13:59:28.204567 6379 catalog_manager.cc:5576] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531) does not have the latest schema for tablet 3bc91db6e866464f89d3e797e976f020 (table rebalancer_dynamic_table [id=4f197d059f224052b2ca8471a151015e]). Expected version 1 got 0
I20240503 13:59:28.206398 10156 tablet.cc:1722] T 3bc91db6e866464f89d3e797e976f020 P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:28.207147 10156 tablet.cc:1620] T 3bc91db6e866464f89d3e797e976f020 P f58065db70b64ef8b1dc499aa6ae4460: MemRowSet was empty: no flush needed.
I20240503 13:59:28.241302 10081 consensus_queue.cc:1035] T 42061f6b71d74b98814092ba97a75086 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:28.246137 10081 consensus_queue.cc:1035] T 42061f6b71d74b98814092ba97a75086 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:28.264741 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:28.270711 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 7de0d3fb7896466d9de4aa642b7bc81d (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=1926a1fe583f4b698c4ba055a9785267]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:28.271064 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7de0d3fb7896466d9de4aa642b7bc81d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:28.271039 9323 tablet_service.cc:1467] Processing CreateTablet for tablet 9ff9b89e4e3247c59ae48ecfeb72f56b (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=1926a1fe583f4b698c4ba055a9785267]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:28.271337 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ff9b89e4e3247c59ae48ecfeb72f56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:28.271479 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 7de0d3fb7896466d9de4aa642b7bc81d (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=1926a1fe583f4b698c4ba055a9785267]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:28.271723 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7de0d3fb7896466d9de4aa642b7bc81d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:28.273200 9191 tablet_service.cc:1467] Processing CreateTablet for tablet 9ff9b89e4e3247c59ae48ecfeb72f56b (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=1926a1fe583f4b698c4ba055a9785267]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:28.273427 9191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ff9b89e4e3247c59ae48ecfeb72f56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:28.273574 10163 tablet_bootstrap.cc:492] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:28.274003 9190 tablet_service.cc:1467] Processing CreateTablet for tablet ee688e32b7d049509a39259d13439cf6 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=1926a1fe583f4b698c4ba055a9785267]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:28.274206 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee688e32b7d049509a39259d13439cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:28.274228 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 7de0d3fb7896466d9de4aa642b7bc81d (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=1926a1fe583f4b698c4ba055a9785267]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:28.274482 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7de0d3fb7896466d9de4aa642b7bc81d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:28.274677 10163 tablet_bootstrap.cc:654] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:28.275882 10165 tablet_bootstrap.cc:492] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:28.276190 10142 tablet_bootstrap.cc:492] T 7de0d3fb7896466d9de4aa642b7bc81d P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:28.277072 10165 tablet_bootstrap.cc:654] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:28.277177 10142 tablet_bootstrap.cc:654] T 7de0d3fb7896466d9de4aa642b7bc81d P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:28.277209 10162 tablet_bootstrap.cc:492] T 7de0d3fb7896466d9de4aa642b7bc81d P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:28.277972 10163 tablet_bootstrap.cc:492] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:28.278056 10163 ts_tablet_manager.cc:1367] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:28.278191 10162 tablet_bootstrap.cc:654] T 7de0d3fb7896466d9de4aa642b7bc81d P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:28.278297 10142 tablet_bootstrap.cc:492] T 7de0d3fb7896466d9de4aa642b7bc81d P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:28.278309 6769 tablet_service.cc:1467] Processing CreateTablet for tablet 9ff9b89e4e3247c59ae48ecfeb72f56b (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=1926a1fe583f4b698c4ba055a9785267]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:28.278390 10142 ts_tablet_manager.cc:1367] T 7de0d3fb7896466d9de4aa642b7bc81d P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:28.278621 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ff9b89e4e3247c59ae48ecfeb72f56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:28.278620 10163 raft_consensus.cc:348] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:28.278725 10163 raft_consensus.cc:374] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:28.278757 10163 raft_consensus.cc:729] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:28.278836 10142 raft_consensus.cc:348] T 7de0d3fb7896466d9de4aa642b7bc81d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:28.278923 10142 raft_consensus.cc:374] T 7de0d3fb7896466d9de4aa642b7bc81d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:28.278951 10142 raft_consensus.cc:729] T 7de0d3fb7896466d9de4aa642b7bc81d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:28.279094 10142 consensus_queue.cc:260] T 7de0d3fb7896466d9de4aa642b7bc81d P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:28.279022 10163 consensus_queue.cc:260] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:28.279419 10142 ts_tablet_manager.cc:1398] T 7de0d3fb7896466d9de4aa642b7bc81d P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:28.279573 10163 ts_tablet_manager.cc:1398] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:28.279666 10162 tablet_bootstrap.cc:492] T 7de0d3fb7896466d9de4aa642b7bc81d P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:28.279745 10162 ts_tablet_manager.cc:1367] T 7de0d3fb7896466d9de4aa642b7bc81d P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:28.280189 10162 raft_consensus.cc:348] T 7de0d3fb7896466d9de4aa642b7bc81d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:28.280282 10162 raft_consensus.cc:374] T 7de0d3fb7896466d9de4aa642b7bc81d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:28.280310 10162 raft_consensus.cc:729] T 7de0d3fb7896466d9de4aa642b7bc81d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:28.280436 10162 consensus_queue.cc:260] T 7de0d3fb7896466d9de4aa642b7bc81d P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:28.280730 10162 ts_tablet_manager.cc:1398] T 7de0d3fb7896466d9de4aa642b7bc81d P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:28.280912 10170 tablet_bootstrap.cc:492] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:28.281885 10170 tablet_bootstrap.cc:654] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:28.283056 10170 tablet_bootstrap.cc:492] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:28.283138 10170 ts_tablet_manager.cc:1367] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:28.283813 10170 raft_consensus.cc:348] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:28.284328 10170 raft_consensus.cc:374] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:28.284390 10170 raft_consensus.cc:729] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:28.284551 10170 consensus_queue.cc:260] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:28.284629 9058 tablet_service.cc:1467] Processing CreateTablet for tablet ee688e32b7d049509a39259d13439cf6 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=1926a1fe583f4b698c4ba055a9785267]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:28.284842 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee688e32b7d049509a39259d13439cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:28.286124 10170 ts_tablet_manager.cc:1398] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:28.286584 10166 tablet_bootstrap.cc:492] T 7de0d3fb7896466d9de4aa642b7bc81d P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:28.287310 10142 tablet_bootstrap.cc:492] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:28.287873 10166 tablet_bootstrap.cc:654] T 7de0d3fb7896466d9de4aa642b7bc81d P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:28.288293 10142 tablet_bootstrap.cc:654] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:28.289633 10142 tablet_bootstrap.cc:492] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:28.289584 9323 tablet_service.cc:1467] Processing CreateTablet for tablet ee688e32b7d049509a39259d13439cf6 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=1926a1fe583f4b698c4ba055a9785267]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:28.289723 10142 ts_tablet_manager.cc:1367] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:28.289872 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee688e32b7d049509a39259d13439cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:28.290191 10142 raft_consensus.cc:348] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:28.290293 10142 raft_consensus.cc:374] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:28.290414 10142 raft_consensus.cc:729] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:28.290585 10142 consensus_queue.cc:260] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:28.291220 10142 ts_tablet_manager.cc:1398] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:28.293151 10163 tablet_bootstrap.cc:492] T ee688e32b7d049509a39259d13439cf6 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:28.294039 10163 tablet_bootstrap.cc:654] T ee688e32b7d049509a39259d13439cf6 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:28.295023 10163 tablet_bootstrap.cc:492] T ee688e32b7d049509a39259d13439cf6 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:28.295122 10163 ts_tablet_manager.cc:1367] T ee688e32b7d049509a39259d13439cf6 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:28.296043 10163 raft_consensus.cc:348] T ee688e32b7d049509a39259d13439cf6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:28.296176 10163 raft_consensus.cc:374] T ee688e32b7d049509a39259d13439cf6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:28.296226 10163 raft_consensus.cc:729] T ee688e32b7d049509a39259d13439cf6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:28.296373 10163 consensus_queue.cc:260] T ee688e32b7d049509a39259d13439cf6 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:28.297127 10163 ts_tablet_manager.cc:1398] T ee688e32b7d049509a39259d13439cf6 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:28.300010 10165 tablet_bootstrap.cc:492] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:28.300113 10165 ts_tablet_manager.cc:1367] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240503 13:59:28.300244 10166 tablet_bootstrap.cc:492] T 7de0d3fb7896466d9de4aa642b7bc81d P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:28.300348 10166 ts_tablet_manager.cc:1367] T 7de0d3fb7896466d9de4aa642b7bc81d P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 13:59:28.300671 10165 raft_consensus.cc:348] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:28.300773 10165 raft_consensus.cc:374] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:28.300814 10165 raft_consensus.cc:729] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:28.300779 10166 raft_consensus.cc:348] T 7de0d3fb7896466d9de4aa642b7bc81d P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:28.300882 10166 raft_consensus.cc:374] T 7de0d3fb7896466d9de4aa642b7bc81d P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:28.300911 10166 raft_consensus.cc:729] T 7de0d3fb7896466d9de4aa642b7bc81d P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:28.300940 10165 consensus_queue.cc:260] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:28.301026 10166 consensus_queue.cc:260] T 7de0d3fb7896466d9de4aa642b7bc81d P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:28.301205 10165 ts_tablet_manager.cc:1398] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:28.301273 10166 ts_tablet_manager.cc:1398] T 7de0d3fb7896466d9de4aa642b7bc81d P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:28.301422 10165 tablet_bootstrap.cc:492] T ee688e32b7d049509a39259d13439cf6 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:28.302337 10165 tablet_bootstrap.cc:654] T ee688e32b7d049509a39259d13439cf6 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:28.303654 10165 tablet_bootstrap.cc:492] T ee688e32b7d049509a39259d13439cf6 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:28.303884 10165 ts_tablet_manager.cc:1367] T ee688e32b7d049509a39259d13439cf6 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:28.304430 10165 raft_consensus.cc:348] T ee688e32b7d049509a39259d13439cf6 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:28.304538 10165 raft_consensus.cc:374] T ee688e32b7d049509a39259d13439cf6 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:28.304564 10165 raft_consensus.cc:729] T ee688e32b7d049509a39259d13439cf6 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:28.304705 10165 consensus_queue.cc:260] T ee688e32b7d049509a39259d13439cf6 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:28.305040 10165 ts_tablet_manager.cc:1398] T ee688e32b7d049509a39259d13439cf6 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:28.307658 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:28.313197 10127 tablet.cc:1722] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:28.313542 10127 tablet.cc:1620] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:28.313652 10156 tablet.cc:1722] T 3bc91db6e866464f89d3e797e976f020 P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:28.313747 10126 tablet.cc:1722] T 3bc91db6e866464f89d3e797e976f020 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:28.313917 10156 tablet.cc:1620] T 3bc91db6e866464f89d3e797e976f020 P f58065db70b64ef8b1dc499aa6ae4460: MemRowSet was empty: no flush needed.
I20240503 13:59:28.314059 10126 tablet.cc:1620] T 3bc91db6e866464f89d3e797e976f020 P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:28.319123 6383 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=4f197d059f224052b2ca8471a151015e] alter complete (version 2)
I20240503 13:59:28.333917 10081 raft_consensus.cc:482] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:28.334020 10081 raft_consensus.cc:504] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:28.334401 10081 leader_election.cc:288] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:28.334740 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff9b89e4e3247c59ae48ecfeb72f56b" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:28.334741 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff9b89e4e3247c59ae48ecfeb72f56b" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:28.334874 6790 raft_consensus.cc:2454] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:28.334874 9343 raft_consensus.cc:2454] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:28.335179 9148 leader_election.cc:302] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:28.335405 10081 raft_consensus.cc:2781] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:28.335472 10081 raft_consensus.cc:482] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:28.335526 10081 raft_consensus.cc:3037] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:28.336436 10081 raft_consensus.cc:504] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:28.336725 10081 leader_election.cc:288] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:28.337050 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff9b89e4e3247c59ae48ecfeb72f56b" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:28.337052 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff9b89e4e3247c59ae48ecfeb72f56b" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:28.337144 6790 raft_consensus.cc:3037] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:28.337150 9343 raft_consensus.cc:3037] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:28.337864 9343 raft_consensus.cc:2454] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:28.337908 6790 raft_consensus.cc:2454] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:28.338179 9146 leader_election.cc:302] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:28.338526 10081 raft_consensus.cc:2781] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:28.338619 10081 raft_consensus.cc:686] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 13:59:28.338753 10081 consensus_queue.cc:237] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:28.340579 6380 catalog_manager.cc:5484] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:28.422013 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:28.426976 10127 tablet.cc:1722] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:28.427392 10126 tablet.cc:1722] T 3bc91db6e866464f89d3e797e976f020 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:28.427469 10156 tablet.cc:1722] T 3bc91db6e866464f89d3e797e976f020 P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:28.432087 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=4f197d059f224052b2ca8471a151015e] alter complete (version 3)
I20240503 13:59:28.503648 10105 raft_consensus.cc:482] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:28.503783 10105 raft_consensus.cc:504] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:28.504168 10105 leader_election.cc:288] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:28.504531 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee688e32b7d049509a39259d13439cf6" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:28.504607 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee688e32b7d049509a39259d13439cf6" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:28.504678 9205 raft_consensus.cc:2454] T ee688e32b7d049509a39259d13439cf6 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:28.504724 9343 raft_consensus.cc:2454] T ee688e32b7d049509a39259d13439cf6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:28.504978 9017 leader_election.cc:302] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:28.505120 10105 raft_consensus.cc:2781] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:28.505178 10105 raft_consensus.cc:482] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:28.505214 10105 raft_consensus.cc:3037] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:28.505997 10105 raft_consensus.cc:504] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:28.506299 10105 leader_election.cc:288] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:28.506650 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee688e32b7d049509a39259d13439cf6" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:28.506625 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee688e32b7d049509a39259d13439cf6" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:28.506765 9343 raft_consensus.cc:3037] T ee688e32b7d049509a39259d13439cf6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:28.506765 9205 raft_consensus.cc:3037] T ee688e32b7d049509a39259d13439cf6 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:28.507725 9205 raft_consensus.cc:2454] T ee688e32b7d049509a39259d13439cf6 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:28.507725 9343 raft_consensus.cc:2454] T ee688e32b7d049509a39259d13439cf6 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:28.508049 9016 leader_election.cc:302] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:28.508208 10105 raft_consensus.cc:2781] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:28.508273 10105 raft_consensus.cc:686] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 13:59:28.508435 10105 consensus_queue.cc:237] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:28.510304 6383 catalog_manager.cc:5484] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:28.515080 9973 raft_consensus.cc:482] T 7de0d3fb7896466d9de4aa642b7bc81d P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:28.515182 9973 raft_consensus.cc:504] T 7de0d3fb7896466d9de4aa642b7bc81d P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:28.515684 9973 leader_election.cc:288] T 7de0d3fb7896466d9de4aa642b7bc81d P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:28.515983 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de0d3fb7896466d9de4aa642b7bc81d" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:28.515992 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de0d3fb7896466d9de4aa642b7bc81d" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:28.516084 9081 raft_consensus.cc:2454] T 7de0d3fb7896466d9de4aa642b7bc81d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:28.516117 6658 raft_consensus.cc:2454] T 7de0d3fb7896466d9de4aa642b7bc81d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:28.516372 6464 leader_election.cc:302] T 7de0d3fb7896466d9de4aa642b7bc81d P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:28.516539 9973 raft_consensus.cc:2781] T 7de0d3fb7896466d9de4aa642b7bc81d P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:28.516598 9973 raft_consensus.cc:482] T 7de0d3fb7896466d9de4aa642b7bc81d P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:28.516633 9973 raft_consensus.cc:3037] T 7de0d3fb7896466d9de4aa642b7bc81d P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:28.517522 9973 raft_consensus.cc:504] T 7de0d3fb7896466d9de4aa642b7bc81d P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:28.517819 9973 leader_election.cc:288] T 7de0d3fb7896466d9de4aa642b7bc81d P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Requested vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:28.518078 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de0d3fb7896466d9de4aa642b7bc81d" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:28.518198 6658 raft_consensus.cc:3037] T 7de0d3fb7896466d9de4aa642b7bc81d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:28.518153 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de0d3fb7896466d9de4aa642b7bc81d" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:28.518273 9081 raft_consensus.cc:3037] T 7de0d3fb7896466d9de4aa642b7bc81d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:28.519191 6658 raft_consensus.cc:2454] T 7de0d3fb7896466d9de4aa642b7bc81d P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:28.519191 9081 raft_consensus.cc:2454] T 7de0d3fb7896466d9de4aa642b7bc81d P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:28.519495 6464 leader_election.cc:302] T 7de0d3fb7896466d9de4aa642b7bc81d P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:28.519678 9973 raft_consensus.cc:2781] T 7de0d3fb7896466d9de4aa642b7bc81d P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:28.519822 9973 raft_consensus.cc:686] T 7de0d3fb7896466d9de4aa642b7bc81d P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Becoming Leader. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Running, Role: LEADER
I20240503 13:59:28.519961 9973 consensus_queue.cc:237] T 7de0d3fb7896466d9de4aa642b7bc81d P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:28.522254 6380 catalog_manager.cc:5484] T 7de0d3fb7896466d9de4aa642b7bc81d P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:28.533674 6380 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:28.533800 6380 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:28.535698 6380 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 3bc91db6e866464f89d3e797e976f020
I20240503 13:59:28.536250 6901 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bc91db6e866464f89d3e797e976f020 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:28 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:28.536262 9058 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bc91db6e866464f89d3e797e976f020 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:28 UTC) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:28.536556 8927 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bc91db6e866464f89d3e797e976f020 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:28 UTC) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:28.536695 10180 tablet_replica.cc:330] T 3bc91db6e866464f89d3e797e976f020 P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:28.536856 10180 raft_consensus.cc:2229] T 3bc91db6e866464f89d3e797e976f020 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:28.536885 10182 tablet_replica.cc:330] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:28.536888 10181 tablet_replica.cc:330] T 3bc91db6e866464f89d3e797e976f020 P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:28.537003 10182 raft_consensus.cc:2229] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:28.537017 10181 raft_consensus.cc:2229] T 3bc91db6e866464f89d3e797e976f020 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:28.537032 10180 raft_consensus.cc:2258] T 3bc91db6e866464f89d3e797e976f020 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:28.537124 10181 raft_consensus.cc:2258] T 3bc91db6e866464f89d3e797e976f020 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:28.537205 10182 raft_consensus.cc:2258] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:28.537617 10180 ts_tablet_manager.cc:1875] T 3bc91db6e866464f89d3e797e976f020 P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:28.537775 10182 ts_tablet_manager.cc:1875] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:28.537791 10181 ts_tablet_manager.cc:1875] T 3bc91db6e866464f89d3e797e976f020 P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:28.539716 10180 ts_tablet_manager.cc:1888] T 3bc91db6e866464f89d3e797e976f020 P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:28.539788 10180 log.cc:1196] T 3bc91db6e866464f89d3e797e976f020 P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/3bc91db6e866464f89d3e797e976f020
I20240503 13:59:28.540124 10182 ts_tablet_manager.cc:1888] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:28.540145 10180 ts_tablet_manager.cc:1909] T 3bc91db6e866464f89d3e797e976f020 P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:28.540189 10181 ts_tablet_manager.cc:1888] T 3bc91db6e866464f89d3e797e976f020 P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:28.540234 10182 log.cc:1196] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/3bc91db6e866464f89d3e797e976f020
I20240503 13:59:28.540275 10181 log.cc:1196] T 3bc91db6e866464f89d3e797e976f020 P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/3bc91db6e866464f89d3e797e976f020
I20240503 13:59:28.540572 10182 ts_tablet_manager.cc:1909] T 3bc91db6e866464f89d3e797e976f020 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting consensus metadata
I20240503 13:59:28.540633 10181 ts_tablet_manager.cc:1909] T 3bc91db6e866464f89d3e797e976f020 P 8d574fb4a89d4eea947c859470d44dd4: Deleting consensus metadata
I20240503 13:59:28.541062 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet 3bc91db6e866464f89d3e797e976f020 (table rebalancer_dynamic_table_new_name [id=4f197d059f224052b2ca8471a151015e]) successfully deleted
I20240503 13:59:28.541247 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet 3bc91db6e866464f89d3e797e976f020 (table rebalancer_dynamic_table_new_name [id=4f197d059f224052b2ca8471a151015e]) successfully deleted
I20240503 13:59:28.541373 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 3bc91db6e866464f89d3e797e976f020 (table rebalancer_dynamic_table_new_name [id=4f197d059f224052b2ca8471a151015e]) successfully deleted
I20240503 13:59:28.637454 6380 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:28.643126 6901 tablet_service.cc:1467] Processing CreateTablet for tablet 8531eb58b00b41b698b132b676e7bab2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9641217ada7344f389a253f9b16ab902]), partition=
I20240503 13:59:28.643265 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 8531eb58b00b41b698b132b676e7bab2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9641217ada7344f389a253f9b16ab902]), partition=
I20240503 13:59:28.643350 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 8531eb58b00b41b698b132b676e7bab2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9641217ada7344f389a253f9b16ab902]), partition=
I20240503 13:59:28.643417 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8531eb58b00b41b698b132b676e7bab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:28.643503 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8531eb58b00b41b698b132b676e7bab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:28.643581 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8531eb58b00b41b698b132b676e7bab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:28.645668 10166 tablet_bootstrap.cc:492] T 8531eb58b00b41b698b132b676e7bab2 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:28.645675 10162 tablet_bootstrap.cc:492] T 8531eb58b00b41b698b132b676e7bab2 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:28.646401 10162 tablet_bootstrap.cc:654] T 8531eb58b00b41b698b132b676e7bab2 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:28.646481 10166 tablet_bootstrap.cc:654] T 8531eb58b00b41b698b132b676e7bab2 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:28.647166 10184 tablet_bootstrap.cc:492] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:28.647630 10166 tablet_bootstrap.cc:492] T 8531eb58b00b41b698b132b676e7bab2 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:28.647715 10166 ts_tablet_manager.cc:1367] T 8531eb58b00b41b698b132b676e7bab2 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:28.648137 10184 tablet_bootstrap.cc:654] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:28.648140 10166 raft_consensus.cc:348] T 8531eb58b00b41b698b132b676e7bab2 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:28.648232 10166 raft_consensus.cc:374] T 8531eb58b00b41b698b132b676e7bab2 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:28.648269 10166 raft_consensus.cc:729] T 8531eb58b00b41b698b132b676e7bab2 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:28.648396 10166 consensus_queue.cc:260] T 8531eb58b00b41b698b132b676e7bab2 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:28.648702 10166 ts_tablet_manager.cc:1398] T 8531eb58b00b41b698b132b676e7bab2 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:28.649921 10162 tablet_bootstrap.cc:492] T 8531eb58b00b41b698b132b676e7bab2 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:28.650012 10162 ts_tablet_manager.cc:1367] T 8531eb58b00b41b698b132b676e7bab2 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:28.650061 10184 tablet_bootstrap.cc:492] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:28.650166 10184 ts_tablet_manager.cc:1367] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:28.650439 10162 raft_consensus.cc:348] T 8531eb58b00b41b698b132b676e7bab2 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:28.650527 10162 raft_consensus.cc:374] T 8531eb58b00b41b698b132b676e7bab2 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:28.650557 10162 raft_consensus.cc:729] T 8531eb58b00b41b698b132b676e7bab2 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:28.650578 10184 raft_consensus.cc:348] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:28.650662 10184 raft_consensus.cc:374] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:28.650691 10184 raft_consensus.cc:729] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:28.650684 10162 consensus_queue.cc:260] T 8531eb58b00b41b698b132b676e7bab2 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:28.650804 10184 consensus_queue.cc:260] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:28.650959 10162 ts_tablet_manager.cc:1398] T 8531eb58b00b41b698b132b676e7bab2 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:28.651667 10184 ts_tablet_manager.cc:1398] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:28.685310 10132 raft_consensus.cc:482] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:28.685429 10132 raft_consensus.cc:504] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:28.685951 10132 leader_election.cc:288] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:28.686321 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8531eb58b00b41b698b132b676e7bab2" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:28.686345 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8531eb58b00b41b698b132b676e7bab2" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:28.686456 6658 raft_consensus.cc:2454] T 8531eb58b00b41b698b132b676e7bab2 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:28.686501 6527 raft_consensus.cc:2454] T 8531eb58b00b41b698b132b676e7bab2 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:28.686887 6857 leader_election.cc:302] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:28.687072 10132 raft_consensus.cc:2781] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:28.687137 10132 raft_consensus.cc:482] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:28.687193 10132 raft_consensus.cc:3037] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:28.688025 10132 raft_consensus.cc:504] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:28.688336 10132 leader_election.cc:288] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:28.688627 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8531eb58b00b41b698b132b676e7bab2" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:28.688663 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8531eb58b00b41b698b132b676e7bab2" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:28.688724 6658 raft_consensus.cc:3037] T 8531eb58b00b41b698b132b676e7bab2 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:28.688745 6527 raft_consensus.cc:3037] T 8531eb58b00b41b698b132b676e7bab2 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:28.689508 6658 raft_consensus.cc:2454] T 8531eb58b00b41b698b132b676e7bab2 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:28.689730 6527 raft_consensus.cc:2454] T 8531eb58b00b41b698b132b676e7bab2 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:28.689997 6857 leader_election.cc:302] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:28.690186 10132 raft_consensus.cc:2781] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:28.690255 10132 raft_consensus.cc:686] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:28.690393 10132 consensus_queue.cc:237] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:28.692204 6383 catalog_manager.cc:5484] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:28.798501 6383 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:28.802814 6527 raft_consensus.cc:1261] T 8531eb58b00b41b698b132b676e7bab2 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:28.802858 6658 raft_consensus.cc:1261] T 8531eb58b00b41b698b132b676e7bab2 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:28.803261 10132 consensus_queue.cc:1035] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:28.803450 10183 consensus_queue.cc:1035] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:28.806095 10156 tablet.cc:1722] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:28.806318 6383 catalog_manager.cc:5576] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119) does not have the latest schema for tablet 8531eb58b00b41b698b132b676e7bab2 (table rebalancer_dynamic_table [id=9641217ada7344f389a253f9b16ab902]). Expected version 1 got 0
I20240503 13:59:28.806491 10156 tablet.cc:1620] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460: MemRowSet was empty: no flush needed.
I20240503 13:59:28.807353 6380 catalog_manager.cc:5576] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021) does not have the latest schema for tablet 8531eb58b00b41b698b132b676e7bab2 (table rebalancer_dynamic_table [id=9641217ada7344f389a253f9b16ab902]). Expected version 1 got 0
I20240503 13:59:28.809249 10195 tablet.cc:1722] T 8531eb58b00b41b698b132b676e7bab2 P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:28.809631 10195 tablet.cc:1620] T 8531eb58b00b41b698b132b676e7bab2 P 2145d15ab2d74c0287839266f1411a43: MemRowSet was empty: no flush needed.
I20240503 13:59:28.810511 6383 catalog_manager.cc:5576] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021) does not have the latest schema for tablet 8531eb58b00b41b698b132b676e7bab2 (table rebalancer_dynamic_table [id=9641217ada7344f389a253f9b16ab902]). Expected version 1 got 0
I20240503 13:59:28.810788 6385 catalog_manager.cc:5576] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119) does not have the latest schema for tablet 8531eb58b00b41b698b132b676e7bab2 (table rebalancer_dynamic_table [id=9641217ada7344f389a253f9b16ab902]). Expected version 1 got 0
I20240503 13:59:28.810747 10194 tablet.cc:1722] T 8531eb58b00b41b698b132b676e7bab2 P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:28.811064 10194 tablet.cc:1620] T 8531eb58b00b41b698b132b676e7bab2 P 5f46e19644fd4f4b825c86ba6af768da: MemRowSet was empty: no flush needed.
I20240503 13:59:28.812101 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=9641217ada7344f389a253f9b16ab902] alter complete (version 1)
I20240503 13:59:28.884486 10198 consensus_queue.cc:1035] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:28.887846 10081 consensus_queue.cc:1035] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:28.914181 6383 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:28.919605 10156 tablet.cc:1722] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:28.919950 10156 tablet.cc:1620] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460: MemRowSet was empty: no flush needed.
I20240503 13:59:28.920224 10195 tablet.cc:1722] T 8531eb58b00b41b698b132b676e7bab2 P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:28.920249 10194 tablet.cc:1722] T 8531eb58b00b41b698b132b676e7bab2 P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:28.920480 10195 tablet.cc:1620] T 8531eb58b00b41b698b132b676e7bab2 P 2145d15ab2d74c0287839266f1411a43: MemRowSet was empty: no flush needed.
I20240503 13:59:28.920564 10194 tablet.cc:1620] T 8531eb58b00b41b698b132b676e7bab2 P 5f46e19644fd4f4b825c86ba6af768da: MemRowSet was empty: no flush needed.
I20240503 13:59:28.926334 6385 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=9641217ada7344f389a253f9b16ab902] alter complete (version 2)
I20240503 13:59:28.961052 10105 consensus_queue.cc:1035] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:28.964644 9987 consensus_queue.cc:1035] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:29.030751 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:29.036545 10156 tablet.cc:1722] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:29.036938 10194 tablet.cc:1722] T 8531eb58b00b41b698b132b676e7bab2 P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:29.037483 10195 tablet.cc:1722] T 8531eb58b00b41b698b132b676e7bab2 P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:29.042671 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=9641217ada7344f389a253f9b16ab902] alter complete (version 3)
I20240503 13:59:29.070932 6380 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:29.078661 8927 tablet_service.cc:1467] Processing CreateTablet for tablet a16acc33ab6c45858c52cceb4355207a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=595e2f47ca074ec4bd181e870ba60ff9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:29.078754 9058 tablet_service.cc:1467] Processing CreateTablet for tablet a16acc33ab6c45858c52cceb4355207a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=595e2f47ca074ec4bd181e870ba60ff9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:29.078969 8927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a16acc33ab6c45858c52cceb4355207a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:29.078974 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a16acc33ab6c45858c52cceb4355207a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:29.078991 9190 tablet_service.cc:1467] Processing CreateTablet for tablet 5fba96f4e9e14a5eb6fda9ed132f4e08 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=595e2f47ca074ec4bd181e870ba60ff9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:29.079231 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fba96f4e9e14a5eb6fda9ed132f4e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:29.079737 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 5fba96f4e9e14a5eb6fda9ed132f4e08 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=595e2f47ca074ec4bd181e870ba60ff9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:29.079970 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fba96f4e9e14a5eb6fda9ed132f4e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:29.081741 10212 tablet_bootstrap.cc:492] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:29.081725 9323 tablet_service.cc:1467] Processing CreateTablet for tablet a16acc33ab6c45858c52cceb4355207a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=595e2f47ca074ec4bd181e870ba60ff9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:29.081983 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a16acc33ab6c45858c52cceb4355207a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:29.082985 10212 tablet_bootstrap.cc:654] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:29.083575 6769 tablet_service.cc:1467] Processing CreateTablet for tablet 5fba96f4e9e14a5eb6fda9ed132f4e08 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=595e2f47ca074ec4bd181e870ba60ff9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:29.083858 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fba96f4e9e14a5eb6fda9ed132f4e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:29.085043 9322 tablet_service.cc:1467] Processing CreateTablet for tablet aecb348d4b3b4500a587c92a7fefab4e (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=595e2f47ca074ec4bd181e870ba60ff9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:29.085279 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aecb348d4b3b4500a587c92a7fefab4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:29.085575 10214 tablet_bootstrap.cc:492] T a16acc33ab6c45858c52cceb4355207a P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:29.085855 10211 tablet_bootstrap.cc:492] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:29.085887 10210 tablet_bootstrap.cc:492] T a16acc33ab6c45858c52cceb4355207a P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:29.086602 10214 tablet_bootstrap.cc:654] T a16acc33ab6c45858c52cceb4355207a P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:29.086855 10211 tablet_bootstrap.cc:654] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:29.086933 10210 tablet_bootstrap.cc:654] T a16acc33ab6c45858c52cceb4355207a P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:29.088454 10211 tablet_bootstrap.cc:492] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:29.088554 10211 ts_tablet_manager.cc:1367] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:29.089017 10211 raft_consensus.cc:348] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:29.089154 10211 raft_consensus.cc:374] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:29.089200 10211 raft_consensus.cc:729] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:29.089337 10211 consensus_queue.cc:260] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:29.089427 10218 tablet_bootstrap.cc:492] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:29.089993 6508 tablet_service.cc:1467] Processing CreateTablet for tablet aecb348d4b3b4500a587c92a7fefab4e (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=595e2f47ca074ec4bd181e870ba60ff9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:29.090276 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aecb348d4b3b4500a587c92a7fefab4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:29.090276 9058 tablet_service.cc:1467] Processing CreateTablet for tablet aecb348d4b3b4500a587c92a7fefab4e (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=595e2f47ca074ec4bd181e870ba60ff9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:29.090379 10218 tablet_bootstrap.cc:654] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:29.090523 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aecb348d4b3b4500a587c92a7fefab4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:29.091137 10210 tablet_bootstrap.cc:492] T a16acc33ab6c45858c52cceb4355207a P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:29.091212 10210 ts_tablet_manager.cc:1367] T a16acc33ab6c45858c52cceb4355207a P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:29.091673 10210 raft_consensus.cc:348] T a16acc33ab6c45858c52cceb4355207a P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:29.091771 10210 raft_consensus.cc:374] T a16acc33ab6c45858c52cceb4355207a P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:29.091802 10210 raft_consensus.cc:729] T a16acc33ab6c45858c52cceb4355207a P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:29.091938 10210 consensus_queue.cc:260] T a16acc33ab6c45858c52cceb4355207a P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:29.092700 10166 tablet_bootstrap.cc:492] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:29.093055 9973 consensus_queue.cc:1035] T 7de0d3fb7896466d9de4aa642b7bc81d P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:29.093564 10210 ts_tablet_manager.cc:1398] T a16acc33ab6c45858c52cceb4355207a P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:29.093721 10210 tablet_bootstrap.cc:492] T aecb348d4b3b4500a587c92a7fefab4e P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:29.093915 10166 tablet_bootstrap.cc:654] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:29.095208 10210 tablet_bootstrap.cc:654] T aecb348d4b3b4500a587c92a7fefab4e P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:29.096130 10210 tablet_bootstrap.cc:492] T aecb348d4b3b4500a587c92a7fefab4e P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:29.096132 10212 tablet_bootstrap.cc:492] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:29.096207 10210 ts_tablet_manager.cc:1367] T aecb348d4b3b4500a587c92a7fefab4e P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:29.096217 10212 ts_tablet_manager.cc:1367] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 13:59:29.096618 10210 raft_consensus.cc:348] T aecb348d4b3b4500a587c92a7fefab4e P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:29.096621 10212 raft_consensus.cc:348] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:29.096709 10212 raft_consensus.cc:374] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:29.096709 10210 raft_consensus.cc:374] T aecb348d4b3b4500a587c92a7fefab4e P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:29.096741 10210 raft_consensus.cc:729] T aecb348d4b3b4500a587c92a7fefab4e P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:29.096741 10212 raft_consensus.cc:729] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:29.096871 10210 consensus_queue.cc:260] T aecb348d4b3b4500a587c92a7fefab4e P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:29.096863 10212 consensus_queue.cc:260] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:29.097154 10212 ts_tablet_manager.cc:1398] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:29.097258 10210 ts_tablet_manager.cc:1398] T aecb348d4b3b4500a587c92a7fefab4e P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:29.097303 10212 tablet_bootstrap.cc:492] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:29.098044 10212 tablet_bootstrap.cc:654] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:29.099812 10211 ts_tablet_manager.cc:1398] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 13:59:29.102507 10166 tablet_bootstrap.cc:492] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:29.102600 10166 ts_tablet_manager.cc:1367] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 13:59:29.103106 10166 raft_consensus.cc:348] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:29.103235 10166 raft_consensus.cc:374] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:29.103578 10214 tablet_bootstrap.cc:492] T a16acc33ab6c45858c52cceb4355207a P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:29.103664 10214 ts_tablet_manager.cc:1367] T a16acc33ab6c45858c52cceb4355207a P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.002s
I20240503 13:59:29.104182 10166 raft_consensus.cc:729] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:29.104176 10214 raft_consensus.cc:348] T a16acc33ab6c45858c52cceb4355207a P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:29.104281 10214 raft_consensus.cc:374] T a16acc33ab6c45858c52cceb4355207a P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:29.104312 10214 raft_consensus.cc:729] T a16acc33ab6c45858c52cceb4355207a P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:29.104333 10166 consensus_queue.cc:260] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:29.104420 10214 consensus_queue.cc:260] T a16acc33ab6c45858c52cceb4355207a P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:29.104611 10166 ts_tablet_manager.cc:1398] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:29.104686 10214 ts_tablet_manager.cc:1398] T a16acc33ab6c45858c52cceb4355207a P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:29.104833 10214 tablet_bootstrap.cc:492] T aecb348d4b3b4500a587c92a7fefab4e P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:29.105823 10214 tablet_bootstrap.cc:654] T aecb348d4b3b4500a587c92a7fefab4e P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:29.108772 10212 tablet_bootstrap.cc:492] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:29.108920 10212 ts_tablet_manager.cc:1367] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 13:59:29.108960 10218 tablet_bootstrap.cc:492] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:29.109071 10218 ts_tablet_manager.cc:1367] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.000s
I20240503 13:59:29.109539 10212 raft_consensus.cc:348] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:29.109599 10218 raft_consensus.cc:348] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:29.109717 10218 raft_consensus.cc:374] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:29.109771 10218 raft_consensus.cc:729] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:29.109751 10212 raft_consensus.cc:374] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:29.109918 10212 raft_consensus.cc:729] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:29.109946 10218 consensus_queue.cc:260] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:29.110284 10218 ts_tablet_manager.cc:1398] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:29.110260 10212 consensus_queue.cc:260] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:29.111373 10214 tablet_bootstrap.cc:492] T aecb348d4b3b4500a587c92a7fefab4e P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:29.111454 10214 ts_tablet_manager.cc:1367] T aecb348d4b3b4500a587c92a7fefab4e P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 13:59:29.111821 10212 ts_tablet_manager.cc:1398] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:29.112407 10214 raft_consensus.cc:348] T aecb348d4b3b4500a587c92a7fefab4e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:29.112524 10214 raft_consensus.cc:374] T aecb348d4b3b4500a587c92a7fefab4e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:29.112562 10214 raft_consensus.cc:729] T aecb348d4b3b4500a587c92a7fefab4e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:29.112753 10214 consensus_queue.cc:260] T aecb348d4b3b4500a587c92a7fefab4e P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:29.113075 10214 ts_tablet_manager.cc:1398] T aecb348d4b3b4500a587c92a7fefab4e P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:29.116681 9973 consensus_queue.cc:1035] T 7de0d3fb7896466d9de4aa642b7bc81d P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:29.145318 6380 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:29.145498 6380 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:29.147943 6380 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 8531eb58b00b41b698b132b676e7bab2
I20240503 13:59:29.148638 6901 tablet_service.cc:1514] Processing DeleteTablet for tablet 8531eb58b00b41b698b132b676e7bab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:29 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:29.148743 6640 tablet_service.cc:1514] Processing DeleteTablet for tablet 8531eb58b00b41b698b132b676e7bab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:29 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:29.148730 6508 tablet_service.cc:1514] Processing DeleteTablet for tablet 8531eb58b00b41b698b132b676e7bab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:29 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:29.149137 10230 tablet_replica.cc:330] T 8531eb58b00b41b698b132b676e7bab2 P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:29.149143 10228 tablet_replica.cc:330] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:29.149363 10230 raft_consensus.cc:2229] T 8531eb58b00b41b698b132b676e7bab2 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:29.149415 10228 raft_consensus.cc:2229] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:29.149559 10229 tablet_replica.cc:330] T 8531eb58b00b41b698b132b676e7bab2 P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:29.149566 10230 raft_consensus.cc:2258] T 8531eb58b00b41b698b132b676e7bab2 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:29.149693 10228 raft_consensus.cc:2258] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:29.149914 10229 raft_consensus.cc:2229] T 8531eb58b00b41b698b132b676e7bab2 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:29.150132 10229 raft_consensus.cc:2258] T 8531eb58b00b41b698b132b676e7bab2 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:29.150270 10230 ts_tablet_manager.cc:1875] T 8531eb58b00b41b698b132b676e7bab2 P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:29.150457 10228 ts_tablet_manager.cc:1875] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:29.151621 10229 ts_tablet_manager.cc:1875] T 8531eb58b00b41b698b132b676e7bab2 P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:29.153501 10228 ts_tablet_manager.cc:1888] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:29.153671 10228 log.cc:1196] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/8531eb58b00b41b698b132b676e7bab2
I20240503 13:59:29.154065 10228 ts_tablet_manager.cc:1909] T 8531eb58b00b41b698b132b676e7bab2 P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:29.154968 10229 ts_tablet_manager.cc:1888] T 8531eb58b00b41b698b132b676e7bab2 P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:29.155117 10229 log.cc:1196] T 8531eb58b00b41b698b132b676e7bab2 P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/8531eb58b00b41b698b132b676e7bab2
I20240503 13:59:29.155447 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet 8531eb58b00b41b698b132b676e7bab2 (table rebalancer_dynamic_table_new_name [id=9641217ada7344f389a253f9b16ab902]) successfully deleted
I20240503 13:59:29.155651 10229 ts_tablet_manager.cc:1909] T 8531eb58b00b41b698b132b676e7bab2 P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:29.155689 10230 ts_tablet_manager.cc:1888] T 8531eb58b00b41b698b132b676e7bab2 P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:29.155827 10230 log.cc:1196] T 8531eb58b00b41b698b132b676e7bab2 P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/8531eb58b00b41b698b132b676e7bab2
I20240503 13:59:29.156196 10230 ts_tablet_manager.cc:1909] T 8531eb58b00b41b698b132b676e7bab2 P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:29.156436 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet 8531eb58b00b41b698b132b676e7bab2 (table rebalancer_dynamic_table_new_name [id=9641217ada7344f389a253f9b16ab902]) successfully deleted
I20240503 13:59:29.157037 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 8531eb58b00b41b698b132b676e7bab2 (table rebalancer_dynamic_table_new_name [id=9641217ada7344f389a253f9b16ab902]) successfully deleted
I20240503 13:59:29.208288 10148 raft_consensus.cc:482] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:29.208431 10148 raft_consensus.cc:504] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:29.208920 10148 leader_election.cc:288] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:29.209239 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16acc33ab6c45858c52cceb4355207a" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:29.209260 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16acc33ab6c45858c52cceb4355207a" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:29.209436 9080 raft_consensus.cc:2454] T a16acc33ab6c45858c52cceb4355207a P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:29.209435 9343 raft_consensus.cc:2454] T a16acc33ab6c45858c52cceb4355207a P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:29.209724 8883 leader_election.cc:302] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:29.209887 10148 raft_consensus.cc:2781] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:29.209933 10148 raft_consensus.cc:482] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:29.209970 10148 raft_consensus.cc:3037] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:29.211140 10148 raft_consensus.cc:504] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:29.211493 10148 leader_election.cc:288] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:29.211841 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16acc33ab6c45858c52cceb4355207a" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:29.211853 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16acc33ab6c45858c52cceb4355207a" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:29.211967 9080 raft_consensus.cc:3037] T a16acc33ab6c45858c52cceb4355207a P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:29.211982 9343 raft_consensus.cc:3037] T a16acc33ab6c45858c52cceb4355207a P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:29.212930 9080 raft_consensus.cc:2454] T a16acc33ab6c45858c52cceb4355207a P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:29.212930 9343 raft_consensus.cc:2454] T a16acc33ab6c45858c52cceb4355207a P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:29.213223 8884 leader_election.cc:302] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 0d77752181e944698e809e325e5937e6; no voters:
I20240503 13:59:29.213456 10148 raft_consensus.cc:2781] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:29.213548 10148 raft_consensus.cc:686] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 13:59:29.213712 10148 consensus_queue.cc:237] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:29.215982 6380 catalog_manager.cc:5484] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:29.250208 6380 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:29.255352 9322 tablet_service.cc:1467] Processing CreateTablet for tablet d53903a652af41b18f892060c4c51959 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a8e33f83d0094960a0e2c5066ac00f6e]), partition=
I20240503 13:59:29.255625 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d53903a652af41b18f892060c4c51959. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:29.257427 10214 tablet_bootstrap.cc:492] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:29.257969 8928 tablet_service.cc:1467] Processing CreateTablet for tablet d53903a652af41b18f892060c4c51959 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a8e33f83d0094960a0e2c5066ac00f6e]), partition=
I20240503 13:59:29.258220 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d53903a652af41b18f892060c4c51959. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:29.258405 10214 tablet_bootstrap.cc:654] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:29.259075 9058 tablet_service.cc:1467] Processing CreateTablet for tablet d53903a652af41b18f892060c4c51959 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a8e33f83d0094960a0e2c5066ac00f6e]), partition=
I20240503 13:59:29.259341 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d53903a652af41b18f892060c4c51959. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:29.260939 10210 tablet_bootstrap.cc:492] T d53903a652af41b18f892060c4c51959 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:29.261551 10212 tablet_bootstrap.cc:492] T d53903a652af41b18f892060c4c51959 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:29.261756 10210 tablet_bootstrap.cc:654] T d53903a652af41b18f892060c4c51959 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:29.261854 10214 tablet_bootstrap.cc:492] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:29.261933 10214 ts_tablet_manager.cc:1367] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 13:59:29.262426 10214 raft_consensus.cc:348] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:29.262555 10214 raft_consensus.cc:374] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:29.262593 10214 raft_consensus.cc:729] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:29.262707 10212 tablet_bootstrap.cc:654] T d53903a652af41b18f892060c4c51959 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:29.262712 10214 consensus_queue.cc:260] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:29.262974 10214 ts_tablet_manager.cc:1398] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:29.264336 10212 tablet_bootstrap.cc:492] T d53903a652af41b18f892060c4c51959 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:29.264428 10212 ts_tablet_manager.cc:1367] T d53903a652af41b18f892060c4c51959 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:29.264971 10212 raft_consensus.cc:348] T d53903a652af41b18f892060c4c51959 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:29.265084 10212 raft_consensus.cc:374] T d53903a652af41b18f892060c4c51959 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:29.265121 10212 raft_consensus.cc:729] T d53903a652af41b18f892060c4c51959 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:29.265252 10212 consensus_queue.cc:260] T d53903a652af41b18f892060c4c51959 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:29.265630 10212 ts_tablet_manager.cc:1398] T d53903a652af41b18f892060c4c51959 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:29.266611 10210 tablet_bootstrap.cc:492] T d53903a652af41b18f892060c4c51959 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:29.266722 10210 ts_tablet_manager.cc:1367] T d53903a652af41b18f892060c4c51959 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 13:59:29.267189 10210 raft_consensus.cc:348] T d53903a652af41b18f892060c4c51959 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:29.267298 10210 raft_consensus.cc:374] T d53903a652af41b18f892060c4c51959 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:29.267343 10210 raft_consensus.cc:729] T d53903a652af41b18f892060c4c51959 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:29.267458 10210 consensus_queue.cc:260] T d53903a652af41b18f892060c4c51959 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:29.267737 10210 ts_tablet_manager.cc:1398] T d53903a652af41b18f892060c4c51959 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:29.292721 10226 raft_consensus.cc:482] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:29.292850 10226 raft_consensus.cc:504] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:29.293335 10226 leader_election.cc:288] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:29.293730 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aecb348d4b3b4500a587c92a7fefab4e" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:29.293879 9080 raft_consensus.cc:2454] T aecb348d4b3b4500a587c92a7fefab4e P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:29.295264 6464 leader_election.cc:302] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:29.295512 10226 raft_consensus.cc:2781] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:29.295565 10226 raft_consensus.cc:482] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:29.295615 10226 raft_consensus.cc:3037] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:29.297086 10226 raft_consensus.cc:504] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:29.297515 10226 leader_election.cc:288] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:29.299088 9342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aecb348d4b3b4500a587c92a7fefab4e" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:29.299230 9342 raft_consensus.cc:3037] T aecb348d4b3b4500a587c92a7fefab4e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:29.300153 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aecb348d4b3b4500a587c92a7fefab4e" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:29.300284 9080 raft_consensus.cc:3037] T aecb348d4b3b4500a587c92a7fefab4e P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:29.300283 9342 raft_consensus.cc:2454] T aecb348d4b3b4500a587c92a7fefab4e P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:29.300640 6465 leader_election.cc:302] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:29.299080 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aecb348d4b3b4500a587c92a7fefab4e" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:29.301000 9343 raft_consensus.cc:2362] T aecb348d4b3b4500a587c92a7fefab4e P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1. Re-sending same reply.
I20240503 13:59:29.301271 9080 raft_consensus.cc:2454] T aecb348d4b3b4500a587c92a7fefab4e P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:29.301394 10226 raft_consensus.cc:2781] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:29.301476 10226 raft_consensus.cc:686] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Becoming Leader. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Running, Role: LEADER
I20240503 13:59:29.301614 10226 consensus_queue.cc:237] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:29.303663 6380 catalog_manager.cc:5484] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:29.330732 10148 raft_consensus.cc:482] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:29.330842 10148 raft_consensus.cc:504] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:29.331336 10148 leader_election.cc:288] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:29.331698 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fba96f4e9e14a5eb6fda9ed132f4e08" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:29.331859 9211 raft_consensus.cc:2454] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:29.331897 10208 raft_consensus.cc:482] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:29.332027 10208 raft_consensus.cc:504] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:29.332160 8883 leader_election.cc:302] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:29.332309 10148 raft_consensus.cc:2781] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:29.332362 10148 raft_consensus.cc:482] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:29.332412 10148 raft_consensus.cc:3037] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:29.332432 10208 leader_election.cc:288] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:29.332737 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fba96f4e9e14a5eb6fda9ed132f4e08" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:29.332854 9211 raft_consensus.cc:2454] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:29.332821 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fba96f4e9e14a5eb6fda9ed132f4e08" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:29.333103 6726 leader_election.cc:302] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:29.333281 10208 raft_consensus.cc:2781] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:29.333336 10208 raft_consensus.cc:482] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:29.333395 10208 raft_consensus.cc:3037] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:29.333557 10148 raft_consensus.cc:504] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:29.333875 10148 leader_election.cc:288] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:29.334002 8947 raft_consensus.cc:2379] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a85e492c8154e9881a9cc2a105d7e64 in current term 1: Already voted for candidate 06ca8dbc004c42b99faa527a4e6f9545 in this term.
I20240503 13:59:29.334477 10208 raft_consensus.cc:504] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:29.334751 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fba96f4e9e14a5eb6fda9ed132f4e08" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:29.334851 9211 raft_consensus.cc:3037] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:29.334864 10208 leader_election.cc:288] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:29.335551 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fba96f4e9e14a5eb6fda9ed132f4e08" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:29.335697 8947 raft_consensus.cc:2379] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a85e492c8154e9881a9cc2a105d7e64 in current term 1: Already voted for candidate 06ca8dbc004c42b99faa527a4e6f9545 in this term.
I20240503 13:59:29.335704 9211 raft_consensus.cc:2454] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:29.335892 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fba96f4e9e14a5eb6fda9ed132f4e08" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:29.336035 9205 raft_consensus.cc:2379] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a85e492c8154e9881a9cc2a105d7e64 in current term 1: Already voted for candidate 06ca8dbc004c42b99faa527a4e6f9545 in this term.
I20240503 13:59:29.336149 8883 leader_election.cc:302] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:29.336364 6726 leader_election.cc:302] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64; no voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735
I20240503 13:59:29.336500 10208 raft_consensus.cc:2726] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 13:59:29.336834 10148 raft_consensus.cc:2781] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:29.336812 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fba96f4e9e14a5eb6fda9ed132f4e08" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:29.336831 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fba96f4e9e14a5eb6fda9ed132f4e08" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:29.336923 10148 raft_consensus.cc:686] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 13:59:29.336982 6790 raft_consensus.cc:2379] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 06ca8dbc004c42b99faa527a4e6f9545 in current term 1: Already voted for candidate 4a85e492c8154e9881a9cc2a105d7e64 in this term.
I20240503 13:59:29.337056 10148 consensus_queue.cc:237] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:29.338896 6380 catalog_manager.cc:5484] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:29.395617 10057 raft_consensus.cc:482] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:29.395730 10057 raft_consensus.cc:504] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:29.396039 10057 leader_election.cc:288] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:29.396417 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d53903a652af41b18f892060c4c51959" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:29.396559 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d53903a652af41b18f892060c4c51959" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:29.396603 8947 raft_consensus.cc:2454] T d53903a652af41b18f892060c4c51959 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:29.396683 9080 raft_consensus.cc:2454] T d53903a652af41b18f892060c4c51959 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:29.396939 9278 leader_election.cc:302] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 0d77752181e944698e809e325e5937e6; no voters:
I20240503 13:59:29.397225 10057 raft_consensus.cc:2781] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:29.397285 10057 raft_consensus.cc:482] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:29.397347 10057 raft_consensus.cc:3037] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:29.398432 10057 raft_consensus.cc:504] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:29.398819 10057 leader_election.cc:288] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:29.399166 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d53903a652af41b18f892060c4c51959" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:29.399211 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d53903a652af41b18f892060c4c51959" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:29.399343 9080 raft_consensus.cc:3037] T d53903a652af41b18f892060c4c51959 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:29.399343 8947 raft_consensus.cc:3037] T d53903a652af41b18f892060c4c51959 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:29.400300 8947 raft_consensus.cc:2454] T d53903a652af41b18f892060c4c51959 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:29.400300 9080 raft_consensus.cc:2454] T d53903a652af41b18f892060c4c51959 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:29.400697 9278 leader_election.cc:302] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:29.401580 10057 raft_consensus.cc:2781] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:29.401696 10057 raft_consensus.cc:686] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:29.401839 10057 consensus_queue.cc:237] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:29.404157 6379 catalog_manager.cc:5484] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:29.514122 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:29.518231 8947 raft_consensus.cc:1261] T d53903a652af41b18f892060c4c51959 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:29.518234 9080 raft_consensus.cc:1261] T d53903a652af41b18f892060c4c51959 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:29.518646 10207 consensus_queue.cc:1035] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:29.518788 10057 consensus_queue.cc:1035] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:29.521960 6379 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet d53903a652af41b18f892060c4c51959 (table rebalancer_dynamic_table [id=a8e33f83d0094960a0e2c5066ac00f6e]). Expected version 1 got 0
I20240503 13:59:29.521993 10248 tablet.cc:1722] T d53903a652af41b18f892060c4c51959 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:29.522325 10248 tablet.cc:1620] T d53903a652af41b18f892060c4c51959 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:29.522784 10247 tablet.cc:1722] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:29.523079 10247 tablet.cc:1620] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:29.523825 6380 catalog_manager.cc:5576] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481) does not have the latest schema for tablet d53903a652af41b18f892060c4c51959 (table rebalancer_dynamic_table [id=a8e33f83d0094960a0e2c5066ac00f6e]). Expected version 1 got 0
I20240503 13:59:29.524874 6386 catalog_manager.cc:5576] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931) does not have the latest schema for tablet d53903a652af41b18f892060c4c51959 (table rebalancer_dynamic_table [id=a8e33f83d0094960a0e2c5066ac00f6e]). Expected version 1 got 0
I20240503 13:59:29.526211 6386 catalog_manager.cc:5576] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481) does not have the latest schema for tablet d53903a652af41b18f892060c4c51959 (table rebalancer_dynamic_table [id=a8e33f83d0094960a0e2c5066ac00f6e]). Expected version 1 got 0
I20240503 13:59:29.526916 10250 tablet.cc:1722] T d53903a652af41b18f892060c4c51959 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:29.527236 10250 tablet.cc:1620] T d53903a652af41b18f892060c4c51959 P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:29.529098 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=a8e33f83d0094960a0e2c5066ac00f6e] alter complete (version 1)
I20240503 13:59:29.633247 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:29.638777 10247 tablet.cc:1722] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:29.639111 10247 tablet.cc:1620] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:29.639437 10250 tablet.cc:1722] T d53903a652af41b18f892060c4c51959 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:29.639815 10250 tablet.cc:1620] T d53903a652af41b18f892060c4c51959 P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:29.640312 10248 tablet.cc:1722] T d53903a652af41b18f892060c4c51959 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:29.640609 10248 tablet.cc:1620] T d53903a652af41b18f892060c4c51959 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:29.644165 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=a8e33f83d0094960a0e2c5066ac00f6e] alter complete (version 2)
I20240503 13:59:29.699879 10091 consensus_queue.cc:1035] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:29.703907 10252 consensus_queue.cc:1035] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:29.746335 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:29.752275 10247 tablet.cc:1722] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:29.752816 10248 tablet.cc:1722] T d53903a652af41b18f892060c4c51959 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:29.753386 10250 tablet.cc:1722] T d53903a652af41b18f892060c4c51959 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:29.754964 10148 consensus_queue.cc:1035] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:29.757478 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=a8e33f83d0094960a0e2c5066ac00f6e] alter complete (version 3)
I20240503 13:59:29.759512 10091 consensus_queue.cc:1035] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:29.862373 6380 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:29.862521 6380 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:29.864638 6380 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet d53903a652af41b18f892060c4c51959
I20240503 13:59:29.865190 9322 tablet_service.cc:1514] Processing DeleteTablet for tablet d53903a652af41b18f892060c4c51959 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:29 UTC) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:29.865224 8928 tablet_service.cc:1514] Processing DeleteTablet for tablet d53903a652af41b18f892060c4c51959 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:29 UTC) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:29.865252 9058 tablet_service.cc:1514] Processing DeleteTablet for tablet d53903a652af41b18f892060c4c51959 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:29 UTC) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:29.865638 10259 tablet_replica.cc:330] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:29.865689 10261 tablet_replica.cc:330] T d53903a652af41b18f892060c4c51959 P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:29.865792 10259 raft_consensus.cc:2229] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:29.865864 10261 raft_consensus.cc:2229] T d53903a652af41b18f892060c4c51959 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:29.865993 10261 raft_consensus.cc:2258] T d53903a652af41b18f892060c4c51959 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:29.866015 10259 raft_consensus.cc:2258] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:29.866288 10260 tablet_replica.cc:330] T d53903a652af41b18f892060c4c51959 P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:29.866449 10260 raft_consensus.cc:2229] T d53903a652af41b18f892060c4c51959 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:29.866595 10260 raft_consensus.cc:2258] T d53903a652af41b18f892060c4c51959 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:29.866683 10261 ts_tablet_manager.cc:1875] T d53903a652af41b18f892060c4c51959 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:29.866783 10259 ts_tablet_manager.cc:1875] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:29.867240 10260 ts_tablet_manager.cc:1875] T d53903a652af41b18f892060c4c51959 P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:29.868966 10259 ts_tablet_manager.cc:1888] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:29.869027 10261 ts_tablet_manager.cc:1888] T d53903a652af41b18f892060c4c51959 P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:29.869087 10259 log.cc:1196] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/d53903a652af41b18f892060c4c51959
I20240503 13:59:29.869118 10261 log.cc:1196] T d53903a652af41b18f892060c4c51959 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/d53903a652af41b18f892060c4c51959
I20240503 13:59:29.869230 10260 ts_tablet_manager.cc:1888] T d53903a652af41b18f892060c4c51959 P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:29.869331 10260 log.cc:1196] T d53903a652af41b18f892060c4c51959 P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/d53903a652af41b18f892060c4c51959
I20240503 13:59:29.869377 10261 ts_tablet_manager.cc:1909] T d53903a652af41b18f892060c4c51959 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting consensus metadata
I20240503 13:59:29.869385 10259 ts_tablet_manager.cc:1909] T d53903a652af41b18f892060c4c51959 P 0d77752181e944698e809e325e5937e6: Deleting consensus metadata
I20240503 13:59:29.869961 10260 ts_tablet_manager.cc:1909] T d53903a652af41b18f892060c4c51959 P 8d574fb4a89d4eea947c859470d44dd4: Deleting consensus metadata
I20240503 13:59:29.869912 10254 consensus_queue.cc:1035] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:29.870349 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet d53903a652af41b18f892060c4c51959 (table rebalancer_dynamic_table_new_name [id=a8e33f83d0094960a0e2c5066ac00f6e]) successfully deleted
I20240503 13:59:29.870682 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet d53903a652af41b18f892060c4c51959 (table rebalancer_dynamic_table_new_name [id=a8e33f83d0094960a0e2c5066ac00f6e]) successfully deleted
I20240503 13:59:29.870827 10254 consensus_queue.cc:1035] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:29.870934 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet d53903a652af41b18f892060c4c51959 (table rebalancer_dynamic_table_new_name [id=a8e33f83d0094960a0e2c5066ac00f6e]) successfully deleted
I20240503 13:59:29.887223 6380 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:29.893065 9058 tablet_service.cc:1467] Processing CreateTablet for tablet b8c8bc1c29934e32a3f30ad20db05462 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=b9c4b58f1b704717873c5fbf98db4e2a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:29.893378 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8c8bc1c29934e32a3f30ad20db05462. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:29.893360 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 3ec64e021d824e15a54c2aa3301f5006 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=b9c4b58f1b704717873c5fbf98db4e2a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:29.893628 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ec64e021d824e15a54c2aa3301f5006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:29.893894 6769 tablet_service.cc:1467] Processing CreateTablet for tablet fdc763e134884bde9e4e66947e5ad126 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=b9c4b58f1b704717873c5fbf98db4e2a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:29.894007 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 3ec64e021d824e15a54c2aa3301f5006 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=b9c4b58f1b704717873c5fbf98db4e2a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:29.894158 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc763e134884bde9e4e66947e5ad126. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:29.894254 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ec64e021d824e15a54c2aa3301f5006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:29.895817 9190 tablet_service.cc:1467] Processing CreateTablet for tablet b8c8bc1c29934e32a3f30ad20db05462 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=b9c4b58f1b704717873c5fbf98db4e2a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:29.896078 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8c8bc1c29934e32a3f30ad20db05462. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:29.896409 9322 tablet_service.cc:1467] Processing CreateTablet for tablet fdc763e134884bde9e4e66947e5ad126 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=b9c4b58f1b704717873c5fbf98db4e2a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:29.896678 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc763e134884bde9e4e66947e5ad126. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:29.897141 10268 tablet_bootstrap.cc:492] T 3ec64e021d824e15a54c2aa3301f5006 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:29.897197 10267 tablet_bootstrap.cc:492] T fdc763e134884bde9e4e66947e5ad126 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:29.898094 10268 tablet_bootstrap.cc:654] T 3ec64e021d824e15a54c2aa3301f5006 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:29.898253 10267 tablet_bootstrap.cc:654] T fdc763e134884bde9e4e66947e5ad126 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:29.898504 6901 tablet_service.cc:1467] Processing CreateTablet for tablet b8c8bc1c29934e32a3f30ad20db05462 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=b9c4b58f1b704717873c5fbf98db4e2a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:29.898757 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8c8bc1c29934e32a3f30ad20db05462. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:29.898989 10272 tablet_bootstrap.cc:492] T fdc763e134884bde9e4e66947e5ad126 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:29.899366 10267 tablet_bootstrap.cc:492] T fdc763e134884bde9e4e66947e5ad126 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:29.899454 10267 ts_tablet_manager.cc:1367] T fdc763e134884bde9e4e66947e5ad126 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:29.899968 10272 tablet_bootstrap.cc:654] T fdc763e134884bde9e4e66947e5ad126 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:29.899930 10267 raft_consensus.cc:348] T fdc763e134884bde9e4e66947e5ad126 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:29.900060 10267 raft_consensus.cc:374] T fdc763e134884bde9e4e66947e5ad126 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:29.900097 10267 raft_consensus.cc:729] T fdc763e134884bde9e4e66947e5ad126 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:29.900214 10267 consensus_queue.cc:260] T fdc763e134884bde9e4e66947e5ad126 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:29.900314 10269 tablet_bootstrap.cc:492] T b8c8bc1c29934e32a3f30ad20db05462 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:29.900554 10267 ts_tablet_manager.cc:1398] T fdc763e134884bde9e4e66947e5ad126 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:29.900869 10265 tablet_bootstrap.cc:492] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:29.902141 9191 tablet_service.cc:1467] Processing CreateTablet for tablet fdc763e134884bde9e4e66947e5ad126 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=b9c4b58f1b704717873c5fbf98db4e2a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:29.902318 10276 tablet_bootstrap.cc:492] T b8c8bc1c29934e32a3f30ad20db05462 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:29.902384 9191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc763e134884bde9e4e66947e5ad126. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:29.902809 9190 tablet_service.cc:1467] Processing CreateTablet for tablet 3ec64e021d824e15a54c2aa3301f5006 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=b9c4b58f1b704717873c5fbf98db4e2a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:29.903008 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ec64e021d824e15a54c2aa3301f5006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:29.903211 10276 tablet_bootstrap.cc:654] T b8c8bc1c29934e32a3f30ad20db05462 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:29.904431 10276 tablet_bootstrap.cc:492] T b8c8bc1c29934e32a3f30ad20db05462 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:29.904508 10276 ts_tablet_manager.cc:1367] T b8c8bc1c29934e32a3f30ad20db05462 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:29.904973 10276 raft_consensus.cc:348] T b8c8bc1c29934e32a3f30ad20db05462 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:29.905159 10276 raft_consensus.cc:374] T b8c8bc1c29934e32a3f30ad20db05462 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:29.905244 10276 raft_consensus.cc:729] T b8c8bc1c29934e32a3f30ad20db05462 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:29.905418 10276 consensus_queue.cc:260] T b8c8bc1c29934e32a3f30ad20db05462 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:29.905791 10276 ts_tablet_manager.cc:1398] T b8c8bc1c29934e32a3f30ad20db05462 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:29.906327 10266 tablet_bootstrap.cc:492] T 3ec64e021d824e15a54c2aa3301f5006 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:29.906641 10265 tablet_bootstrap.cc:654] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:29.907210 10269 tablet_bootstrap.cc:654] T b8c8bc1c29934e32a3f30ad20db05462 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:29.907227 10266 tablet_bootstrap.cc:654] T 3ec64e021d824e15a54c2aa3301f5006 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:29.908350 10268 tablet_bootstrap.cc:492] T 3ec64e021d824e15a54c2aa3301f5006 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:29.908459 10268 ts_tablet_manager.cc:1367] T 3ec64e021d824e15a54c2aa3301f5006 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 13:59:29.908910 10268 raft_consensus.cc:348] T 3ec64e021d824e15a54c2aa3301f5006 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:29.908991 10268 raft_consensus.cc:374] T 3ec64e021d824e15a54c2aa3301f5006 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:29.909018 10268 raft_consensus.cc:729] T 3ec64e021d824e15a54c2aa3301f5006 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:29.909143 10268 consensus_queue.cc:260] T 3ec64e021d824e15a54c2aa3301f5006 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:29.909409 10268 ts_tablet_manager.cc:1398] T 3ec64e021d824e15a54c2aa3301f5006 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:29.909960 10266 tablet_bootstrap.cc:492] T 3ec64e021d824e15a54c2aa3301f5006 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:29.910054 10266 ts_tablet_manager.cc:1367] T 3ec64e021d824e15a54c2aa3301f5006 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:59:29.910773 10265 tablet_bootstrap.cc:492] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:29.910774 10266 raft_consensus.cc:348] T 3ec64e021d824e15a54c2aa3301f5006 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:29.910914 10265 ts_tablet_manager.cc:1367] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 13:59:29.910991 10266 raft_consensus.cc:374] T 3ec64e021d824e15a54c2aa3301f5006 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:29.911051 10266 raft_consensus.cc:729] T 3ec64e021d824e15a54c2aa3301f5006 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:29.911207 10266 consensus_queue.cc:260] T 3ec64e021d824e15a54c2aa3301f5006 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:29.911554 10266 ts_tablet_manager.cc:1398] T 3ec64e021d824e15a54c2aa3301f5006 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:29.911528 10265 raft_consensus.cc:348] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:29.912144 10265 raft_consensus.cc:374] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:29.912285 10265 raft_consensus.cc:729] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:29.912750 10269 tablet_bootstrap.cc:492] T b8c8bc1c29934e32a3f30ad20db05462 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:29.912837 10269 ts_tablet_manager.cc:1367] T b8c8bc1c29934e32a3f30ad20db05462 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240503 13:59:29.913205 10269 raft_consensus.cc:348] T b8c8bc1c29934e32a3f30ad20db05462 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:29.913280 10269 raft_consensus.cc:374] T b8c8bc1c29934e32a3f30ad20db05462 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:29.913300 10269 raft_consensus.cc:729] T b8c8bc1c29934e32a3f30ad20db05462 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:29.913399 10269 consensus_queue.cc:260] T b8c8bc1c29934e32a3f30ad20db05462 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:29.913621 10269 ts_tablet_manager.cc:1398] T b8c8bc1c29934e32a3f30ad20db05462 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:29.913758 10269 tablet_bootstrap.cc:492] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:29.915320 10269 tablet_bootstrap.cc:654] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:29.914850 10265 consensus_queue.cc:260] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:29.916174 10265 ts_tablet_manager.cc:1398] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:59:29.917548 10269 tablet_bootstrap.cc:492] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:29.917564 10272 tablet_bootstrap.cc:492] T fdc763e134884bde9e4e66947e5ad126 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:29.917660 10269 ts_tablet_manager.cc:1367] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:29.917665 10272 ts_tablet_manager.cc:1367] T fdc763e134884bde9e4e66947e5ad126 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.002s
I20240503 13:59:29.918102 10272 raft_consensus.cc:348] T fdc763e134884bde9e4e66947e5ad126 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:29.918124 10269 raft_consensus.cc:348] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:29.918226 10272 raft_consensus.cc:374] T fdc763e134884bde9e4e66947e5ad126 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:29.918224 10269 raft_consensus.cc:374] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:29.918258 10269 raft_consensus.cc:729] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:29.918258 10272 raft_consensus.cc:729] T fdc763e134884bde9e4e66947e5ad126 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:29.918391 10272 consensus_queue.cc:260] T fdc763e134884bde9e4e66947e5ad126 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:29.918413 10269 consensus_queue.cc:260] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:29.918668 10269 ts_tablet_manager.cc:1398] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:29.918841 10269 tablet_bootstrap.cc:492] T 3ec64e021d824e15a54c2aa3301f5006 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:29.919819 10269 tablet_bootstrap.cc:654] T 3ec64e021d824e15a54c2aa3301f5006 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:29.919821 10272 ts_tablet_manager.cc:1398] T fdc763e134884bde9e4e66947e5ad126 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:29.921890 10269 tablet_bootstrap.cc:492] T 3ec64e021d824e15a54c2aa3301f5006 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:29.921955 10269 ts_tablet_manager.cc:1367] T 3ec64e021d824e15a54c2aa3301f5006 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:29.922271 10269 raft_consensus.cc:348] T 3ec64e021d824e15a54c2aa3301f5006 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:29.922350 10269 raft_consensus.cc:374] T 3ec64e021d824e15a54c2aa3301f5006 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:29.922371 10269 raft_consensus.cc:729] T 3ec64e021d824e15a54c2aa3301f5006 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:29.922510 10269 consensus_queue.cc:260] T 3ec64e021d824e15a54c2aa3301f5006 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:29.922799 10269 ts_tablet_manager.cc:1398] T 3ec64e021d824e15a54c2aa3301f5006 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:29.966179 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:29.971261 9322 tablet_service.cc:1467] Processing CreateTablet for tablet 1884351036eb43428e76502cb4402c86 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5146ac405c904d77b4a34f18c8bf5874]), partition=
I20240503 13:59:29.971261 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 1884351036eb43428e76502cb4402c86 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5146ac405c904d77b4a34f18c8bf5874]), partition=
I20240503 13:59:29.971333 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 1884351036eb43428e76502cb4402c86 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5146ac405c904d77b4a34f18c8bf5874]), partition=
I20240503 13:59:29.971544 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1884351036eb43428e76502cb4402c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:29.971554 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1884351036eb43428e76502cb4402c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:29.971601 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1884351036eb43428e76502cb4402c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:29.973503 10268 tablet_bootstrap.cc:492] T 1884351036eb43428e76502cb4402c86 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:29.973604 10272 tablet_bootstrap.cc:492] T 1884351036eb43428e76502cb4402c86 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:29.973986 10284 tablet_bootstrap.cc:492] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:29.974452 10268 tablet_bootstrap.cc:654] T 1884351036eb43428e76502cb4402c86 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:29.974941 10272 tablet_bootstrap.cc:654] T 1884351036eb43428e76502cb4402c86 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:29.975073 10284 tablet_bootstrap.cc:654] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:29.976111 10272 tablet_bootstrap.cc:492] T 1884351036eb43428e76502cb4402c86 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:29.976208 10272 ts_tablet_manager.cc:1367] T 1884351036eb43428e76502cb4402c86 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:29.976208 10268 tablet_bootstrap.cc:492] T 1884351036eb43428e76502cb4402c86 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:29.976297 10268 ts_tablet_manager.cc:1367] T 1884351036eb43428e76502cb4402c86 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:29.976682 10268 raft_consensus.cc:348] T 1884351036eb43428e76502cb4402c86 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:29.976774 10268 raft_consensus.cc:374] T 1884351036eb43428e76502cb4402c86 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:29.976796 10268 raft_consensus.cc:729] T 1884351036eb43428e76502cb4402c86 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:29.976923 10268 consensus_queue.cc:260] T 1884351036eb43428e76502cb4402c86 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:29.977190 10272 raft_consensus.cc:348] T 1884351036eb43428e76502cb4402c86 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:29.977236 10268 ts_tablet_manager.cc:1398] T 1884351036eb43428e76502cb4402c86 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:29.977286 10272 raft_consensus.cc:374] T 1884351036eb43428e76502cb4402c86 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:29.977336 10272 raft_consensus.cc:729] T 1884351036eb43428e76502cb4402c86 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:29.977491 10272 consensus_queue.cc:260] T 1884351036eb43428e76502cb4402c86 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:29.977737 10272 ts_tablet_manager.cc:1398] T 1884351036eb43428e76502cb4402c86 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:29.978230 10284 tablet_bootstrap.cc:492] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:29.978327 10284 ts_tablet_manager.cc:1367] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:29.978863 10284 raft_consensus.cc:348] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:29.978998 10284 raft_consensus.cc:374] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:29.979053 10284 raft_consensus.cc:729] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:29.979224 10284 consensus_queue.cc:260] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:29.979626 10284 ts_tablet_manager.cc:1398] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:29.995898 10198 raft_consensus.cc:482] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:29.996027 10198 raft_consensus.cc:504] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:29.996428 10198 leader_election.cc:288] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:29.996773 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc763e134884bde9e4e66947e5ad126" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:29.996887 9343 raft_consensus.cc:2454] T fdc763e134884bde9e4e66947e5ad126 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:29.996999 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc763e134884bde9e4e66947e5ad126" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:29.997155 6790 raft_consensus.cc:2454] T fdc763e134884bde9e4e66947e5ad126 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:29.997192 9148 leader_election.cc:302] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:29.997704 10198 raft_consensus.cc:2781] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:29.997772 10198 raft_consensus.cc:482] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:29.997807 10198 raft_consensus.cc:3037] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:29.998872 10198 raft_consensus.cc:504] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:29.999208 10198 leader_election.cc:288] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:29.999564 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc763e134884bde9e4e66947e5ad126" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:29.999694 6790 raft_consensus.cc:3037] T fdc763e134884bde9e4e66947e5ad126 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:29.999792 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc763e134884bde9e4e66947e5ad126" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:29.999890 9343 raft_consensus.cc:3037] T fdc763e134884bde9e4e66947e5ad126 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:30.000521 6790 raft_consensus.cc:2454] T fdc763e134884bde9e4e66947e5ad126 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:30.000730 9343 raft_consensus.cc:2454] T fdc763e134884bde9e4e66947e5ad126 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:30.000926 9146 leader_election.cc:302] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:30.001256 10198 raft_consensus.cc:2781] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:30.001353 10198 raft_consensus.cc:686] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 13:59:30.001464 10198 consensus_queue.cc:237] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:30.003201 6379 catalog_manager.cc:5484] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:30.013494 10198 raft_consensus.cc:482] T 3ec64e021d824e15a54c2aa3301f5006 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:30.013597 10198 raft_consensus.cc:504] T 3ec64e021d824e15a54c2aa3301f5006 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:30.014051 10198 leader_election.cc:288] T 3ec64e021d824e15a54c2aa3301f5006 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:30.014214 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec64e021d824e15a54c2aa3301f5006" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:30.014338 6528 raft_consensus.cc:2454] T 3ec64e021d824e15a54c2aa3301f5006 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:30.014384 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec64e021d824e15a54c2aa3301f5006" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:30.014531 6658 raft_consensus.cc:2454] T 3ec64e021d824e15a54c2aa3301f5006 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:30.014654 9147 leader_election.cc:302] T 3ec64e021d824e15a54c2aa3301f5006 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:30.014813 10198 raft_consensus.cc:2781] T 3ec64e021d824e15a54c2aa3301f5006 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:30.014858 10198 raft_consensus.cc:482] T 3ec64e021d824e15a54c2aa3301f5006 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:30.014891 10198 raft_consensus.cc:3037] T 3ec64e021d824e15a54c2aa3301f5006 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:30.015822 10198 raft_consensus.cc:504] T 3ec64e021d824e15a54c2aa3301f5006 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:30.016085 10198 leader_election.cc:288] T 3ec64e021d824e15a54c2aa3301f5006 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:30.016362 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec64e021d824e15a54c2aa3301f5006" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:30.016515 6528 raft_consensus.cc:3037] T 3ec64e021d824e15a54c2aa3301f5006 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:30.017374 6528 raft_consensus.cc:2454] T 3ec64e021d824e15a54c2aa3301f5006 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:30.017619 9147 leader_election.cc:302] T 3ec64e021d824e15a54c2aa3301f5006 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:30.017591 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec64e021d824e15a54c2aa3301f5006" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:30.017711 6658 raft_consensus.cc:3037] T 3ec64e021d824e15a54c2aa3301f5006 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:30.018400 6658 raft_consensus.cc:2454] T 3ec64e021d824e15a54c2aa3301f5006 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:30.018551 10198 raft_consensus.cc:2781] T 3ec64e021d824e15a54c2aa3301f5006 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:30.018637 10198 raft_consensus.cc:686] T 3ec64e021d824e15a54c2aa3301f5006 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 13:59:30.018769 10198 consensus_queue.cc:237] T 3ec64e021d824e15a54c2aa3301f5006 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:30.020407 6379 catalog_manager.cc:5484] T 3ec64e021d824e15a54c2aa3301f5006 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:30.025606 10091 raft_consensus.cc:482] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:30.025717 10091 raft_consensus.cc:504] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:30.026104 10091 leader_election.cc:288] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:30.026424 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1884351036eb43428e76502cb4402c86" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:30.026445 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1884351036eb43428e76502cb4402c86" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:30.026566 6658 raft_consensus.cc:2454] T 1884351036eb43428e76502cb4402c86 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:30.026561 9343 raft_consensus.cc:2454] T 1884351036eb43428e76502cb4402c86 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:30.026829 8884 leader_election.cc:302] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 0d77752181e944698e809e325e5937e6; no voters:
I20240503 13:59:30.026978 10091 raft_consensus.cc:2781] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:30.027035 10091 raft_consensus.cc:482] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:30.027084 10091 raft_consensus.cc:3037] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:30.027869 10091 raft_consensus.cc:504] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:30.028152 10091 leader_election.cc:288] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:30.028460 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1884351036eb43428e76502cb4402c86" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:30.028487 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1884351036eb43428e76502cb4402c86" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:30.028569 9343 raft_consensus.cc:3037] T 1884351036eb43428e76502cb4402c86 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:30.028597 6658 raft_consensus.cc:3037] T 1884351036eb43428e76502cb4402c86 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:30.029282 9343 raft_consensus.cc:2454] T 1884351036eb43428e76502cb4402c86 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:30.029477 6658 raft_consensus.cc:2454] T 1884351036eb43428e76502cb4402c86 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:30.029536 8884 leader_election.cc:302] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 0d77752181e944698e809e325e5937e6; no voters:
I20240503 13:59:30.030241 10091 raft_consensus.cc:2781] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:30.030321 10091 raft_consensus.cc:686] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 13:59:30.030445 10091 consensus_queue.cc:237] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:30.032500 6379 catalog_manager.cc:5484] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:30.143349 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:30.146972 9343 raft_consensus.cc:1261] T 1884351036eb43428e76502cb4402c86 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:30.147384 10252 consensus_queue.cc:1035] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:30.147554 6658 raft_consensus.cc:1261] T 1884351036eb43428e76502cb4402c86 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:30.147867 10252 consensus_queue.cc:1035] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:30.149605 10248 tablet.cc:1722] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:30.149837 6379 catalog_manager.cc:5576] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931) does not have the latest schema for tablet 1884351036eb43428e76502cb4402c86 (table rebalancer_dynamic_table [id=5146ac405c904d77b4a34f18c8bf5874]). Expected version 1 got 0
I20240503 13:59:30.149937 10248 tablet.cc:1620] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:30.150177 10247 tablet.cc:1722] T 1884351036eb43428e76502cb4402c86 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:30.150489 10247 tablet.cc:1620] T 1884351036eb43428e76502cb4402c86 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:30.152810 6380 catalog_manager.cc:5576] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119) does not have the latest schema for tablet 1884351036eb43428e76502cb4402c86 (table rebalancer_dynamic_table [id=5146ac405c904d77b4a34f18c8bf5874]). Expected version 1 got 0
I20240503 13:59:30.154561 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=5146ac405c904d77b4a34f18c8bf5874] alter complete (version 1)
I20240503 13:59:30.155920 10295 tablet.cc:1722] T 1884351036eb43428e76502cb4402c86 P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:30.156213 10295 tablet.cc:1620] T 1884351036eb43428e76502cb4402c86 P 2145d15ab2d74c0287839266f1411a43: MemRowSet was empty: no flush needed.
I20240503 13:59:30.200759 10105 raft_consensus.cc:482] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:30.200871 10105 raft_consensus.cc:504] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:30.201222 10105 leader_election.cc:288] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:30.201583 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c8bc1c29934e32a3f30ad20db05462" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:30.201671 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c8bc1c29934e32a3f30ad20db05462" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:30.201725 9211 raft_consensus.cc:2454] T b8c8bc1c29934e32a3f30ad20db05462 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:30.201830 6922 raft_consensus.cc:2454] T b8c8bc1c29934e32a3f30ad20db05462 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:30.202039 9016 leader_election.cc:302] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:30.202222 10105 raft_consensus.cc:2781] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:30.202278 10105 raft_consensus.cc:482] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:30.202329 10105 raft_consensus.cc:3037] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:30.203181 10105 raft_consensus.cc:504] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:30.203514 10105 leader_election.cc:288] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:30.203816 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c8bc1c29934e32a3f30ad20db05462" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:30.203895 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c8bc1c29934e32a3f30ad20db05462" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:30.203989 6922 raft_consensus.cc:3037] T b8c8bc1c29934e32a3f30ad20db05462 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:30.204030 9211 raft_consensus.cc:3037] T b8c8bc1c29934e32a3f30ad20db05462 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:30.204690 6922 raft_consensus.cc:2454] T b8c8bc1c29934e32a3f30ad20db05462 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:30.204743 9211 raft_consensus.cc:2454] T b8c8bc1c29934e32a3f30ad20db05462 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:30.204977 9016 leader_election.cc:302] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:30.205121 10105 raft_consensus.cc:2781] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:30.205309 10105 raft_consensus.cc:686] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 13:59:30.205446 10105 consensus_queue.cc:237] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:30.207489 6379 catalog_manager.cc:5484] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:30.259205 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:30.265267 10248 tablet.cc:1722] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:30.265712 10248 tablet.cc:1620] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:30.266265 10247 tablet.cc:1722] T 1884351036eb43428e76502cb4402c86 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:30.266556 10247 tablet.cc:1620] T 1884351036eb43428e76502cb4402c86 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:30.269521 10295 tablet.cc:1722] T 1884351036eb43428e76502cb4402c86 P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:30.269959 10295 tablet.cc:1620] T 1884351036eb43428e76502cb4402c86 P 2145d15ab2d74c0287839266f1411a43: MemRowSet was empty: no flush needed.
I20240503 13:59:30.271912 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=5146ac405c904d77b4a34f18c8bf5874] alter complete (version 2)
I20240503 13:59:30.375332 6383 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:30.380484 10248 tablet.cc:1722] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:30.380903 10247 tablet.cc:1722] T 1884351036eb43428e76502cb4402c86 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:30.381008 10295 tablet.cc:1722] T 1884351036eb43428e76502cb4402c86 P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:30.385377 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=5146ac405c904d77b4a34f18c8bf5874] alter complete (version 3)
I20240503 13:59:30.437136 10198 consensus_queue.cc:1035] T 3ec64e021d824e15a54c2aa3301f5006 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:30.441200 10198 consensus_queue.cc:1035] T 3ec64e021d824e15a54c2aa3301f5006 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:30.484231 10198 consensus_queue.cc:1035] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:30.487129 6379 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:30.487321 6379 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:30.490563 6379 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 1884351036eb43428e76502cb4402c86
I20240503 13:59:30.491268 8928 tablet_service.cc:1514] Processing DeleteTablet for tablet 1884351036eb43428e76502cb4402c86 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:30 UTC) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:30.491525 10198 consensus_queue.cc:1035] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:30.491811 9322 tablet_service.cc:1514] Processing DeleteTablet for tablet 1884351036eb43428e76502cb4402c86 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:30 UTC) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:30.491961 6640 tablet_service.cc:1514] Processing DeleteTablet for tablet 1884351036eb43428e76502cb4402c86 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:30 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:30.492416 10306 tablet_replica.cc:330] T 1884351036eb43428e76502cb4402c86 P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:30.492897 10305 tablet_replica.cc:330] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:30.492712 10306 raft_consensus.cc:2229] T 1884351036eb43428e76502cb4402c86 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:30.493077 10305 raft_consensus.cc:2229] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:30.493324 10305 raft_consensus.cc:2258] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:30.493096 10306 raft_consensus.cc:2258] T 1884351036eb43428e76502cb4402c86 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:30.492552 10307 tablet_replica.cc:330] T 1884351036eb43428e76502cb4402c86 P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:30.493783 10307 raft_consensus.cc:2229] T 1884351036eb43428e76502cb4402c86 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:30.493928 10307 raft_consensus.cc:2258] T 1884351036eb43428e76502cb4402c86 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:30.494035 10305 ts_tablet_manager.cc:1875] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:30.494617 10306 ts_tablet_manager.cc:1875] T 1884351036eb43428e76502cb4402c86 P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:30.495878 10307 ts_tablet_manager.cc:1875] T 1884351036eb43428e76502cb4402c86 P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:30.496608 10305 ts_tablet_manager.cc:1888] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:30.496739 10305 log.cc:1196] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/1884351036eb43428e76502cb4402c86
I20240503 13:59:30.497085 10305 ts_tablet_manager.cc:1909] T 1884351036eb43428e76502cb4402c86 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting consensus metadata
I20240503 13:59:30.497833 10307 ts_tablet_manager.cc:1888] T 1884351036eb43428e76502cb4402c86 P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:30.497851 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 1884351036eb43428e76502cb4402c86 (table rebalancer_dynamic_table_new_name [id=5146ac405c904d77b4a34f18c8bf5874]) successfully deleted
I20240503 13:59:30.497906 10307 log.cc:1196] T 1884351036eb43428e76502cb4402c86 P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/1884351036eb43428e76502cb4402c86
I20240503 13:59:30.498194 10307 ts_tablet_manager.cc:1909] T 1884351036eb43428e76502cb4402c86 P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:30.498863 10306 ts_tablet_manager.cc:1888] T 1884351036eb43428e76502cb4402c86 P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:30.498952 10306 log.cc:1196] T 1884351036eb43428e76502cb4402c86 P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/1884351036eb43428e76502cb4402c86
I20240503 13:59:30.499006 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet 1884351036eb43428e76502cb4402c86 (table rebalancer_dynamic_table_new_name [id=5146ac405c904d77b4a34f18c8bf5874]) successfully deleted
I20240503 13:59:30.499271 10306 ts_tablet_manager.cc:1909] T 1884351036eb43428e76502cb4402c86 P 0d77752181e944698e809e325e5937e6: Deleting consensus metadata
I20240503 13:59:30.500063 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet 1884351036eb43428e76502cb4402c86 (table rebalancer_dynamic_table_new_name [id=5146ac405c904d77b4a34f18c8bf5874]) successfully deleted
I20240503 13:59:30.592139 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:30.597720 9322 tablet_service.cc:1467] Processing CreateTablet for tablet 9e78d1fb68424fba9882be211529c79c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a01ffb52ad03406ea6b3a13d312d3c3c]), partition=
I20240503 13:59:30.597782 6901 tablet_service.cc:1467] Processing CreateTablet for tablet 9e78d1fb68424fba9882be211529c79c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a01ffb52ad03406ea6b3a13d312d3c3c]), partition=
I20240503 13:59:30.598019 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e78d1fb68424fba9882be211529c79c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:30.598040 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e78d1fb68424fba9882be211529c79c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:30.600354 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 9e78d1fb68424fba9882be211529c79c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a01ffb52ad03406ea6b3a13d312d3c3c]), partition=
I20240503 13:59:30.600611 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e78d1fb68424fba9882be211529c79c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:30.600926 10311 tablet_bootstrap.cc:492] T 9e78d1fb68424fba9882be211529c79c P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:30.601717 10311 tablet_bootstrap.cc:654] T 9e78d1fb68424fba9882be211529c79c P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:30.602787 10311 tablet_bootstrap.cc:492] T 9e78d1fb68424fba9882be211529c79c P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:30.602836 10310 tablet_bootstrap.cc:492] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:30.602886 10311 ts_tablet_manager.cc:1367] T 9e78d1fb68424fba9882be211529c79c P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:30.603276 10311 raft_consensus.cc:348] T 9e78d1fb68424fba9882be211529c79c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:30.603519 10311 raft_consensus.cc:374] T 9e78d1fb68424fba9882be211529c79c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:30.603602 10311 raft_consensus.cc:729] T 9e78d1fb68424fba9882be211529c79c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:30.603714 10313 tablet_bootstrap.cc:492] T 9e78d1fb68424fba9882be211529c79c P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:30.603812 10310 tablet_bootstrap.cc:654] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:30.603835 10311 consensus_queue.cc:260] T 9e78d1fb68424fba9882be211529c79c P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:30.604233 10311 ts_tablet_manager.cc:1398] T 9e78d1fb68424fba9882be211529c79c P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:30.605214 10313 tablet_bootstrap.cc:654] T 9e78d1fb68424fba9882be211529c79c P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:30.605315 10310 tablet_bootstrap.cc:492] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:30.605408 10310 ts_tablet_manager.cc:1367] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:30.606209 10310 raft_consensus.cc:348] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:30.606393 10310 raft_consensus.cc:374] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:30.606477 10310 raft_consensus.cc:729] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:30.606642 10310 consensus_queue.cc:260] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:30.607833 10313 tablet_bootstrap.cc:492] T 9e78d1fb68424fba9882be211529c79c P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:30.607976 10313 ts_tablet_manager.cc:1367] T 9e78d1fb68424fba9882be211529c79c P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:30.608462 10310 ts_tablet_manager.cc:1398] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:30.608426 10313 raft_consensus.cc:348] T 9e78d1fb68424fba9882be211529c79c P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:30.608521 10313 raft_consensus.cc:374] T 9e78d1fb68424fba9882be211529c79c P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:30.608604 10313 raft_consensus.cc:729] T 9e78d1fb68424fba9882be211529c79c P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:30.608731 10313 consensus_queue.cc:260] T 9e78d1fb68424fba9882be211529c79c P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:30.609006 10313 ts_tablet_manager.cc:1398] T 9e78d1fb68424fba9882be211529c79c P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:30.645527 10105 consensus_queue.cc:1035] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:30.649420 10296 consensus_queue.cc:1035] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:30.691763 10249 raft_consensus.cc:482] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:30.691877 10249 raft_consensus.cc:504] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:30.692353 10249 leader_election.cc:288] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:30.693071 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e78d1fb68424fba9882be211529c79c" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:30.693243 6528 raft_consensus.cc:2454] T 9e78d1fb68424fba9882be211529c79c P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:30.693531 9278 leader_election.cc:302] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:30.693727 10249 raft_consensus.cc:2781] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:30.693799 10249 raft_consensus.cc:482] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:30.693848 10249 raft_consensus.cc:3037] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:30.694840 10249 raft_consensus.cc:504] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:30.695192 10249 leader_election.cc:288] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:30.695578 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e78d1fb68424fba9882be211529c79c" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:30.695719 6528 raft_consensus.cc:3037] T 9e78d1fb68424fba9882be211529c79c P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:30.696645 6528 raft_consensus.cc:2454] T 9e78d1fb68424fba9882be211529c79c P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:30.696956 9278 leader_election.cc:302] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:30.697124 10249 raft_consensus.cc:2781] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:30.697216 10249 raft_consensus.cc:686] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:30.697232 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e78d1fb68424fba9882be211529c79c" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:30.697340 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e78d1fb68424fba9882be211529c79c" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:30.697374 6922 raft_consensus.cc:2454] T 9e78d1fb68424fba9882be211529c79c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:30.697356 10249 consensus_queue.cc:237] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:30.697454 6920 raft_consensus.cc:3037] T 9e78d1fb68424fba9882be211529c79c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:30.698518 6920 raft_consensus.cc:2454] T 9e78d1fb68424fba9882be211529c79c P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:30.699442 6379 catalog_manager.cc:5484] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:30.763378 6379 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:30.769925 9190 tablet_service.cc:1467] Processing CreateTablet for tablet 2ddf3f4cf2884b8785b216a59a5b40f9 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=0d58a21c125546e9acb619279319e0f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:30.770277 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ddf3f4cf2884b8785b216a59a5b40f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:30.770601 6769 tablet_service.cc:1467] Processing CreateTablet for tablet c8b2d8a9ed5f4b09812a85445a8e3b85 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=0d58a21c125546e9acb619279319e0f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:30.770900 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8b2d8a9ed5f4b09812a85445a8e3b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:30.771575 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 2ddf3f4cf2884b8785b216a59a5b40f9 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=0d58a21c125546e9acb619279319e0f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:30.771600 9322 tablet_service.cc:1467] Processing CreateTablet for tablet 2ddf3f4cf2884b8785b216a59a5b40f9 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=0d58a21c125546e9acb619279319e0f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:30.771863 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ddf3f4cf2884b8785b216a59a5b40f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:30.771879 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ddf3f4cf2884b8785b216a59a5b40f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:30.773262 10323 tablet_bootstrap.cc:492] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:30.773936 10310 tablet_bootstrap.cc:492] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:30.774227 6901 tablet_service.cc:1467] Processing CreateTablet for tablet 25b76c3f53624134bdca9f6a093bb51c (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=0d58a21c125546e9acb619279319e0f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:30.774287 10323 tablet_bootstrap.cc:654] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:30.774186 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 25b76c3f53624134bdca9f6a093bb51c (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=0d58a21c125546e9acb619279319e0f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:30.774493 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25b76c3f53624134bdca9f6a093bb51c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:30.774509 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25b76c3f53624134bdca9f6a093bb51c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:30.775146 10310 tablet_bootstrap.cc:654] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:30.776926 9323 tablet_service.cc:1467] Processing CreateTablet for tablet c8b2d8a9ed5f4b09812a85445a8e3b85 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=0d58a21c125546e9acb619279319e0f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:30.777017 10310 tablet_bootstrap.cc:492] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:30.777105 10310 ts_tablet_manager.cc:1367] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 13:59:30.777163 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8b2d8a9ed5f4b09812a85445a8e3b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:30.777595 10310 raft_consensus.cc:348] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:30.777704 10311 tablet_bootstrap.cc:492] T 25b76c3f53624134bdca9f6a093bb51c P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:30.777717 10310 raft_consensus.cc:374] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:30.777755 10310 raft_consensus.cc:729] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:30.777894 10310 consensus_queue.cc:260] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:30.778136 10310 ts_tablet_manager.cc:1398] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:30.778702 10311 tablet_bootstrap.cc:654] T 25b76c3f53624134bdca9f6a093bb51c P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:30.779173 10310 tablet_bootstrap.cc:492] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:30.779745 8927 tablet_service.cc:1467] Processing CreateTablet for tablet 25b76c3f53624134bdca9f6a093bb51c (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=0d58a21c125546e9acb619279319e0f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:30.780002 8927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25b76c3f53624134bdca9f6a093bb51c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:30.780146 10310 tablet_bootstrap.cc:654] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:30.780301 10311 tablet_bootstrap.cc:492] T 25b76c3f53624134bdca9f6a093bb51c P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:30.780391 10311 ts_tablet_manager.cc:1367] T 25b76c3f53624134bdca9f6a093bb51c P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:30.780910 8928 tablet_service.cc:1467] Processing CreateTablet for tablet c8b2d8a9ed5f4b09812a85445a8e3b85 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=0d58a21c125546e9acb619279319e0f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:30.780900 10311 raft_consensus.cc:348] T 25b76c3f53624134bdca9f6a093bb51c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:30.781024 10311 raft_consensus.cc:374] T 25b76c3f53624134bdca9f6a093bb51c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:30.781056 10311 raft_consensus.cc:729] T 25b76c3f53624134bdca9f6a093bb51c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:30.781172 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8b2d8a9ed5f4b09812a85445a8e3b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:30.781203 10311 consensus_queue.cc:260] T 25b76c3f53624134bdca9f6a093bb51c P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:30.781529 10311 ts_tablet_manager.cc:1398] T 25b76c3f53624134bdca9f6a093bb51c P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:30.782320 10330 tablet_bootstrap.cc:492] T 25b76c3f53624134bdca9f6a093bb51c P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:30.783501 10330 tablet_bootstrap.cc:654] T 25b76c3f53624134bdca9f6a093bb51c P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:30.786125 10324 tablet_bootstrap.cc:492] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:30.786880 10332 tablet_bootstrap.cc:492] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:30.787274 10327 tablet_bootstrap.cc:492] T 25b76c3f53624134bdca9f6a093bb51c P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:30.787475 10324 tablet_bootstrap.cc:654] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:30.788072 10310 tablet_bootstrap.cc:492] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:30.788184 10332 tablet_bootstrap.cc:654] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:30.788225 10310 ts_tablet_manager.cc:1367] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:59:30.788810 10310 raft_consensus.cc:348] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:30.788981 10310 raft_consensus.cc:374] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:30.789058 10310 raft_consensus.cc:729] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:30.789309 10310 consensus_queue.cc:260] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:30.789577 10324 tablet_bootstrap.cc:492] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:30.789680 10310 ts_tablet_manager.cc:1398] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:30.789762 10324 ts_tablet_manager.cc:1367] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:30.789804 10332 tablet_bootstrap.cc:492] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:30.789891 10332 ts_tablet_manager.cc:1367] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:30.790336 10324 raft_consensus.cc:348] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:30.790354 10332 raft_consensus.cc:348] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:30.790441 10324 raft_consensus.cc:374] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:30.790457 10332 raft_consensus.cc:374] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:30.790472 10324 raft_consensus.cc:729] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:30.790490 10332 raft_consensus.cc:729] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:30.790616 10324 consensus_queue.cc:260] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:30.790612 10332 consensus_queue.cc:260] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:30.790977 10332 ts_tablet_manager.cc:1398] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:30.790988 10327 tablet_bootstrap.cc:654] T 25b76c3f53624134bdca9f6a093bb51c P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:30.791016 10324 ts_tablet_manager.cc:1398] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:30.792212 10330 tablet_bootstrap.cc:492] T 25b76c3f53624134bdca9f6a093bb51c P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:30.792315 10330 ts_tablet_manager.cc:1367] T 25b76c3f53624134bdca9f6a093bb51c P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20240503 13:59:30.792407 10323 tablet_bootstrap.cc:492] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:30.792501 10323 ts_tablet_manager.cc:1367] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 13:59:30.792829 10330 raft_consensus.cc:348] T 25b76c3f53624134bdca9f6a093bb51c P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:30.792943 10330 raft_consensus.cc:374] T 25b76c3f53624134bdca9f6a093bb51c P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:30.792979 10330 raft_consensus.cc:729] T 25b76c3f53624134bdca9f6a093bb51c P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:30.793113 10330 consensus_queue.cc:260] T 25b76c3f53624134bdca9f6a093bb51c P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:30.793448 10330 ts_tablet_manager.cc:1398] T 25b76c3f53624134bdca9f6a093bb51c P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:30.793534 10323 raft_consensus.cc:348] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:30.793612 10330 tablet_bootstrap.cc:492] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:30.793620 10323 raft_consensus.cc:374] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:30.793645 10323 raft_consensus.cc:729] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:30.793759 10323 consensus_queue.cc:260] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:30.794085 10323 ts_tablet_manager.cc:1398] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:30.794518 10330 tablet_bootstrap.cc:654] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:30.795779 10327 tablet_bootstrap.cc:492] T 25b76c3f53624134bdca9f6a093bb51c P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:30.795876 10327 ts_tablet_manager.cc:1367] T 25b76c3f53624134bdca9f6a093bb51c P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:59:30.796388 10327 raft_consensus.cc:348] T 25b76c3f53624134bdca9f6a093bb51c P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:30.796571 10327 raft_consensus.cc:374] T 25b76c3f53624134bdca9f6a093bb51c P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:30.796670 10327 raft_consensus.cc:729] T 25b76c3f53624134bdca9f6a093bb51c P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:30.796859 10327 consensus_queue.cc:260] T 25b76c3f53624134bdca9f6a093bb51c P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:30.796989 10330 tablet_bootstrap.cc:492] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:30.797101 10330 ts_tablet_manager.cc:1367] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 13:59:30.797228 10327 ts_tablet_manager.cc:1398] T 25b76c3f53624134bdca9f6a093bb51c P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:30.797546 10330 raft_consensus.cc:348] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:30.797636 10330 raft_consensus.cc:374] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:30.797664 10330 raft_consensus.cc:729] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:30.797823 10330 consensus_queue.cc:260] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:30.798099 10330 ts_tablet_manager.cc:1398] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:30.823096 6383 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:30.827114 6528 raft_consensus.cc:1261] T 9e78d1fb68424fba9882be211529c79c P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:30.827114 6920 raft_consensus.cc:1261] T 9e78d1fb68424fba9882be211529c79c P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:30.827672 10298 consensus_queue.cc:1035] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:30.827993 10249 consensus_queue.cc:1035] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:30.830574 6380 catalog_manager.cc:5576] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531) does not have the latest schema for tablet 9e78d1fb68424fba9882be211529c79c (table rebalancer_dynamic_table [id=a01ffb52ad03406ea6b3a13d312d3c3c]). Expected version 1 got 0
I20240503 13:59:30.831158 6383 catalog_manager.cc:5576] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021) does not have the latest schema for tablet 9e78d1fb68424fba9882be211529c79c (table rebalancer_dynamic_table [id=a01ffb52ad03406ea6b3a13d312d3c3c]). Expected version 1 got 0
I20240503 13:59:30.831518 10247 tablet.cc:1722] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:30.831907 10247 tablet.cc:1620] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:30.833060 10341 tablet.cc:1722] T 9e78d1fb68424fba9882be211529c79c P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:30.833509 10341 tablet.cc:1620] T 9e78d1fb68424fba9882be211529c79c P 5f46e19644fd4f4b825c86ba6af768da: MemRowSet was empty: no flush needed.
I20240503 13:59:30.836045 6383 catalog_manager.cc:5576] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531) does not have the latest schema for tablet 9e78d1fb68424fba9882be211529c79c (table rebalancer_dynamic_table [id=a01ffb52ad03406ea6b3a13d312d3c3c]). Expected version 1 got 0
I20240503 13:59:30.836655 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=a01ffb52ad03406ea6b3a13d312d3c3c] alter complete (version 1)
I20240503 13:59:30.837049 10132 raft_consensus.cc:482] T 25b76c3f53624134bdca9f6a093bb51c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:30.837147 10132 raft_consensus.cc:504] T 25b76c3f53624134bdca9f6a093bb51c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:30.837210 10340 tablet.cc:1722] T 9e78d1fb68424fba9882be211529c79c P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:30.837502 10340 tablet.cc:1620] T 9e78d1fb68424fba9882be211529c79c P f58065db70b64ef8b1dc499aa6ae4460: MemRowSet was empty: no flush needed.
I20240503 13:59:30.837533 10132 leader_election.cc:288] T 25b76c3f53624134bdca9f6a093bb51c P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:30.837882 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b76c3f53624134bdca9f6a093bb51c" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:30.838016 6658 raft_consensus.cc:2454] T 25b76c3f53624134bdca9f6a093bb51c P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:30.838361 6858 leader_election.cc:302] T 25b76c3f53624134bdca9f6a093bb51c P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:30.838547 10132 raft_consensus.cc:2781] T 25b76c3f53624134bdca9f6a093bb51c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:30.838498 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b76c3f53624134bdca9f6a093bb51c" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:30.838596 10132 raft_consensus.cc:482] T 25b76c3f53624134bdca9f6a093bb51c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:30.838631 10132 raft_consensus.cc:3037] T 25b76c3f53624134bdca9f6a093bb51c P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:30.838658 8947 raft_consensus.cc:2454] T 25b76c3f53624134bdca9f6a093bb51c P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:30.839695 10132 raft_consensus.cc:504] T 25b76c3f53624134bdca9f6a093bb51c P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:30.840044 10132 leader_election.cc:288] T 25b76c3f53624134bdca9f6a093bb51c P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:30.840637 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b76c3f53624134bdca9f6a093bb51c" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:30.840663 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b76c3f53624134bdca9f6a093bb51c" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:30.840788 8947 raft_consensus.cc:3037] T 25b76c3f53624134bdca9f6a093bb51c P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:30.840781 6658 raft_consensus.cc:3037] T 25b76c3f53624134bdca9f6a093bb51c P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:30.841625 8947 raft_consensus.cc:2454] T 25b76c3f53624134bdca9f6a093bb51c P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:30.841912 6857 leader_election.cc:302] T 25b76c3f53624134bdca9f6a093bb51c P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:30.841930 6658 raft_consensus.cc:2454] T 25b76c3f53624134bdca9f6a093bb51c P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:30.842069 10342 raft_consensus.cc:2781] T 25b76c3f53624134bdca9f6a093bb51c P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:30.842197 10342 raft_consensus.cc:686] T 25b76c3f53624134bdca9f6a093bb51c P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:30.842341 10342 consensus_queue.cc:237] T 25b76c3f53624134bdca9f6a093bb51c P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:30.845495 6383 catalog_manager.cc:5484] T 25b76c3f53624134bdca9f6a093bb51c P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:30.865370 10298 raft_consensus.cc:482] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:30.865505 10298 raft_consensus.cc:504] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:30.865895 10298 leader_election.cc:288] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:30.866289 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ddf3f4cf2884b8785b216a59a5b40f9" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:30.866312 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ddf3f4cf2884b8785b216a59a5b40f9" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:30.866436 9211 raft_consensus.cc:2454] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:30.866487 9080 raft_consensus.cc:2454] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:30.866825 9278 leader_election.cc:302] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:30.866998 10298 raft_consensus.cc:2781] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:30.867070 10298 raft_consensus.cc:482] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:30.867123 10298 raft_consensus.cc:3037] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:30.868332 10298 raft_consensus.cc:504] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:30.868625 10298 leader_election.cc:288] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:30.868968 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ddf3f4cf2884b8785b216a59a5b40f9" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:30.869009 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ddf3f4cf2884b8785b216a59a5b40f9" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:30.869096 9211 raft_consensus.cc:3037] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:30.869096 9080 raft_consensus.cc:3037] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:30.870016 9080 raft_consensus.cc:2454] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:30.870016 9211 raft_consensus.cc:2454] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:30.870312 9278 leader_election.cc:302] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:30.870465 10298 raft_consensus.cc:2781] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:30.870570 10298 raft_consensus.cc:686] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:30.870712 10298 consensus_queue.cc:237] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:30.872545 6379 catalog_manager.cc:5484] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:30.881734 10273 raft_consensus.cc:482] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:30.881835 10273 raft_consensus.cc:504] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:30.882213 10273 leader_election.cc:288] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:30.882568 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b2d8a9ed5f4b09812a85445a8e3b85" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:30.882686 9343 raft_consensus.cc:2454] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:30.882863 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b2d8a9ed5f4b09812a85445a8e3b85" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:30.882972 6727 leader_election.cc:302] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:30.883005 8947 raft_consensus.cc:2454] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:30.883145 10273 raft_consensus.cc:2781] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:30.883208 10273 raft_consensus.cc:482] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:30.883249 10273 raft_consensus.cc:3037] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:30.884192 10273 raft_consensus.cc:504] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:30.884491 10273 leader_election.cc:288] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:30.884903 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b2d8a9ed5f4b09812a85445a8e3b85" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:30.885020 9343 raft_consensus.cc:3037] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:30.885040 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b2d8a9ed5f4b09812a85445a8e3b85" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:30.885160 8947 raft_consensus.cc:3037] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:30.885825 9343 raft_consensus.cc:2454] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:30.885975 8947 raft_consensus.cc:2454] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:30.886086 6727 leader_election.cc:302] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:30.886260 10273 raft_consensus.cc:2781] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:30.886422 10273 raft_consensus.cc:686] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:30.886579 10273 consensus_queue.cc:237] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:30.888783 6379 catalog_manager.cc:5484] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:30.944653 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:30.950138 10247 tablet.cc:1722] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:30.950627 10247 tablet.cc:1620] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:30.950893 10341 tablet.cc:1722] T 9e78d1fb68424fba9882be211529c79c P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:30.950875 10340 tablet.cc:1722] T 9e78d1fb68424fba9882be211529c79c P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:30.951154 10340 tablet.cc:1620] T 9e78d1fb68424fba9882be211529c79c P f58065db70b64ef8b1dc499aa6ae4460: MemRowSet was empty: no flush needed.
I20240503 13:59:30.951153 10341 tablet.cc:1620] T 9e78d1fb68424fba9882be211529c79c P 5f46e19644fd4f4b825c86ba6af768da: MemRowSet was empty: no flush needed.
I20240503 13:59:30.956279 6386 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=a01ffb52ad03406ea6b3a13d312d3c3c] alter complete (version 2)
I20240503 13:59:31.057698 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:31.063393 10247 tablet.cc:1722] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:31.064082 10341 tablet.cc:1722] T 9e78d1fb68424fba9882be211529c79c P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:31.064502 10340 tablet.cc:1722] T 9e78d1fb68424fba9882be211529c79c P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:31.068974 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=a01ffb52ad03406ea6b3a13d312d3c3c] alter complete (version 3)
I20240503 13:59:31.170919 6383 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:31.171098 6383 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:31.173120 6383 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 9e78d1fb68424fba9882be211529c79c
I20240503 13:59:31.173661 9323 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e78d1fb68424fba9882be211529c79c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:31 UTC) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:31.173811 6508 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e78d1fb68424fba9882be211529c79c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:31 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:31.173811 6901 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e78d1fb68424fba9882be211529c79c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:31 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:31.174010 10347 tablet_replica.cc:330] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:31.174108 10349 tablet_replica.cc:330] T 9e78d1fb68424fba9882be211529c79c P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:31.174141 10348 tablet_replica.cc:330] T 9e78d1fb68424fba9882be211529c79c P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:31.174161 10347 raft_consensus.cc:2229] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:31.174263 10349 raft_consensus.cc:2229] T 9e78d1fb68424fba9882be211529c79c P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:31.174294 10348 raft_consensus.cc:2229] T 9e78d1fb68424fba9882be211529c79c P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:31.174384 10347 raft_consensus.cc:2258] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:31.174393 10349 raft_consensus.cc:2258] T 9e78d1fb68424fba9882be211529c79c P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:31.174401 10348 raft_consensus.cc:2258] T 9e78d1fb68424fba9882be211529c79c P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:31.175027 10349 ts_tablet_manager.cc:1875] T 9e78d1fb68424fba9882be211529c79c P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:31.175019 10347 ts_tablet_manager.cc:1875] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:31.175374 10348 ts_tablet_manager.cc:1875] T 9e78d1fb68424fba9882be211529c79c P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:31.177155 10349 ts_tablet_manager.cc:1888] T 9e78d1fb68424fba9882be211529c79c P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:31.177248 10349 log.cc:1196] T 9e78d1fb68424fba9882be211529c79c P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/9e78d1fb68424fba9882be211529c79c
I20240503 13:59:31.177573 10348 ts_tablet_manager.cc:1888] T 9e78d1fb68424fba9882be211529c79c P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:31.177603 10349 ts_tablet_manager.cc:1909] T 9e78d1fb68424fba9882be211529c79c P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:31.177673 10348 log.cc:1196] T 9e78d1fb68424fba9882be211529c79c P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/9e78d1fb68424fba9882be211529c79c
I20240503 13:59:31.178047 10348 ts_tablet_manager.cc:1909] T 9e78d1fb68424fba9882be211529c79c P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:31.178268 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 9e78d1fb68424fba9882be211529c79c (table rebalancer_dynamic_table_new_name [id=a01ffb52ad03406ea6b3a13d312d3c3c]) successfully deleted
I20240503 13:59:31.178660 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet 9e78d1fb68424fba9882be211529c79c (table rebalancer_dynamic_table_new_name [id=a01ffb52ad03406ea6b3a13d312d3c3c]) successfully deleted
I20240503 13:59:31.178893 10347 ts_tablet_manager.cc:1888] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:31.178959 10347 log.cc:1196] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/9e78d1fb68424fba9882be211529c79c
I20240503 13:59:31.179217 10347 ts_tablet_manager.cc:1909] T 9e78d1fb68424fba9882be211529c79c P 0d77752181e944698e809e325e5937e6: Deleting consensus metadata
I20240503 13:59:31.180249 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet 9e78d1fb68424fba9882be211529c79c (table rebalancer_dynamic_table_new_name [id=a01ffb52ad03406ea6b3a13d312d3c3c]) successfully deleted
I20240503 13:59:31.274677 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:31.279960 9190 tablet_service.cc:1467] Processing CreateTablet for tablet 4bfa080bdf784197814aa52b25439638 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b8fddf3bb54a4b46a29445a5a018c8e4]), partition=
I20240503 13:59:31.279981 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 4bfa080bdf784197814aa52b25439638 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b8fddf3bb54a4b46a29445a5a018c8e4]), partition=
I20240503 13:59:31.279981 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 4bfa080bdf784197814aa52b25439638 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b8fddf3bb54a4b46a29445a5a018c8e4]), partition=
I20240503 13:59:31.280236 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bfa080bdf784197814aa52b25439638. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:31.280236 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bfa080bdf784197814aa52b25439638. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:31.280244 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bfa080bdf784197814aa52b25439638. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:31.282135 10332 tablet_bootstrap.cc:492] T 4bfa080bdf784197814aa52b25439638 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:31.282299 10324 tablet_bootstrap.cc:492] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:31.282836 10351 tablet_bootstrap.cc:492] T 4bfa080bdf784197814aa52b25439638 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:31.283098 10332 tablet_bootstrap.cc:654] T 4bfa080bdf784197814aa52b25439638 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:31.283176 10324 tablet_bootstrap.cc:654] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:31.283630 10351 tablet_bootstrap.cc:654] T 4bfa080bdf784197814aa52b25439638 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:31.284314 10332 tablet_bootstrap.cc:492] T 4bfa080bdf784197814aa52b25439638 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:31.284396 10332 ts_tablet_manager.cc:1367] T 4bfa080bdf784197814aa52b25439638 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:31.284566 10324 tablet_bootstrap.cc:492] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:31.284685 10324 ts_tablet_manager.cc:1367] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:31.284863 10332 raft_consensus.cc:348] T 4bfa080bdf784197814aa52b25439638 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:31.284963 10332 raft_consensus.cc:374] T 4bfa080bdf784197814aa52b25439638 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:31.284924 10342 consensus_queue.cc:1035] T 25b76c3f53624134bdca9f6a093bb51c P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:31.284996 10332 raft_consensus.cc:729] T 4bfa080bdf784197814aa52b25439638 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:31.285117 10332 consensus_queue.cc:260] T 4bfa080bdf784197814aa52b25439638 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:31.285189 10324 raft_consensus.cc:348] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:31.285297 10324 raft_consensus.cc:374] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:31.285333 10324 raft_consensus.cc:729] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:31.285434 10332 ts_tablet_manager.cc:1398] T 4bfa080bdf784197814aa52b25439638 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:31.285447 10324 consensus_queue.cc:260] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:31.285749 10324 ts_tablet_manager.cc:1398] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:31.288502 10132 consensus_queue.cc:1035] T 25b76c3f53624134bdca9f6a093bb51c P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:31.291644 10351 tablet_bootstrap.cc:492] T 4bfa080bdf784197814aa52b25439638 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:31.292110 10351 ts_tablet_manager.cc:1367] T 4bfa080bdf784197814aa52b25439638 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:59:31.292531 10351 raft_consensus.cc:348] T 4bfa080bdf784197814aa52b25439638 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:31.292711 10351 raft_consensus.cc:374] T 4bfa080bdf784197814aa52b25439638 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:31.293020 10351 raft_consensus.cc:729] T 4bfa080bdf784197814aa52b25439638 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:31.293183 10351 consensus_queue.cc:260] T 4bfa080bdf784197814aa52b25439638 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:31.293658 10351 ts_tablet_manager.cc:1398] T 4bfa080bdf784197814aa52b25439638 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:31.307030 10298 consensus_queue.cc:1035] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:31.310523 10339 consensus_queue.cc:1035] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:31.321697 10273 consensus_queue.cc:1035] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:31.325161 10345 consensus_queue.cc:1035] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:31.334345 10350 raft_consensus.cc:482] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:31.334444 10350 raft_consensus.cc:504] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:31.334831 10350 leader_election.cc:288] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:31.335166 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bfa080bdf784197814aa52b25439638" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:31.335166 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bfa080bdf784197814aa52b25439638" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:31.335328 6528 raft_consensus.cc:2454] T 4bfa080bdf784197814aa52b25439638 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:31.335328 9211 raft_consensus.cc:2454] T 4bfa080bdf784197814aa52b25439638 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:31.335604 9016 leader_election.cc:302] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:31.335784 10350 raft_consensus.cc:2781] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:31.335896 10350 raft_consensus.cc:482] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:31.335970 10350 raft_consensus.cc:3037] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:31.337260 10350 raft_consensus.cc:504] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:31.337582 10350 leader_election.cc:288] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:31.337860 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bfa080bdf784197814aa52b25439638" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:31.337919 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bfa080bdf784197814aa52b25439638" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:31.337977 6528 raft_consensus.cc:3037] T 4bfa080bdf784197814aa52b25439638 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:31.338039 9211 raft_consensus.cc:3037] T 4bfa080bdf784197814aa52b25439638 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:31.338847 9211 raft_consensus.cc:2454] T 4bfa080bdf784197814aa52b25439638 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:31.339072 6528 raft_consensus.cc:2454] T 4bfa080bdf784197814aa52b25439638 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:31.339140 9016 leader_election.cc:302] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:31.339349 10350 raft_consensus.cc:2781] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:31.339453 10350 raft_consensus.cc:686] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 13:59:31.339661 10350 consensus_queue.cc:237] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:31.341678 6380 catalog_manager.cc:5484] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:31.407871 6380 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:31.414188 9190 tablet_service.cc:1467] Processing CreateTablet for tablet b08947e5ddd04e7e8dddfa59ca65f713 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=0f264ddfb2274c6488da071d02a13a8e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:31.414336 6769 tablet_service.cc:1467] Processing CreateTablet for tablet b08947e5ddd04e7e8dddfa59ca65f713 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=0f264ddfb2274c6488da071d02a13a8e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:31.414535 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b08947e5ddd04e7e8dddfa59ca65f713. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:31.414589 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b08947e5ddd04e7e8dddfa59ca65f713. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:31.415205 6640 tablet_service.cc:1467] Processing CreateTablet for tablet ea4e399a51f24da699e7cd2264de3c0a (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=0f264ddfb2274c6488da071d02a13a8e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:31.415494 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea4e399a51f24da699e7cd2264de3c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:31.416371 10332 tablet_bootstrap.cc:492] T b08947e5ddd04e7e8dddfa59ca65f713 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:31.416622 9323 tablet_service.cc:1467] Processing CreateTablet for tablet 292ed55b431847f39d832ebe61e6c5e4 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=0f264ddfb2274c6488da071d02a13a8e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:31.416893 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 292ed55b431847f39d832ebe61e6c5e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:31.417338 10332 tablet_bootstrap.cc:654] T b08947e5ddd04e7e8dddfa59ca65f713 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:31.417577 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 292ed55b431847f39d832ebe61e6c5e4 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=0f264ddfb2274c6488da071d02a13a8e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:31.417867 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 292ed55b431847f39d832ebe61e6c5e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:31.417965 9061 tablet_service.cc:1467] Processing CreateTablet for tablet ea4e399a51f24da699e7cd2264de3c0a (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=0f264ddfb2274c6488da071d02a13a8e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:31.418154 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea4e399a51f24da699e7cd2264de3c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:31.418902 10372 tablet_bootstrap.cc:492] T 292ed55b431847f39d832ebe61e6c5e4 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:31.419720 6901 tablet_service.cc:1467] Processing CreateTablet for tablet b08947e5ddd04e7e8dddfa59ca65f713 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=0f264ddfb2274c6488da071d02a13a8e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:31.419917 10372 tablet_bootstrap.cc:654] T 292ed55b431847f39d832ebe61e6c5e4 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:31.420002 10324 tablet_bootstrap.cc:492] T ea4e399a51f24da699e7cd2264de3c0a P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:31.420003 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b08947e5ddd04e7e8dddfa59ca65f713. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:31.420274 9190 tablet_service.cc:1467] Processing CreateTablet for tablet ea4e399a51f24da699e7cd2264de3c0a (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=0f264ddfb2274c6488da071d02a13a8e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:31.420511 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea4e399a51f24da699e7cd2264de3c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:31.421123 10324 tablet_bootstrap.cc:654] T ea4e399a51f24da699e7cd2264de3c0a P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:31.419735 6902 tablet_service.cc:1467] Processing CreateTablet for tablet 292ed55b431847f39d832ebe61e6c5e4 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=0f264ddfb2274c6488da071d02a13a8e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:31.422709 10332 tablet_bootstrap.cc:492] T b08947e5ddd04e7e8dddfa59ca65f713 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:31.422776 6902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 292ed55b431847f39d832ebe61e6c5e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:31.422791 10332 ts_tablet_manager.cc:1367] T b08947e5ddd04e7e8dddfa59ca65f713 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240503 13:59:31.422936 10324 tablet_bootstrap.cc:492] T ea4e399a51f24da699e7cd2264de3c0a P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:31.423012 10324 ts_tablet_manager.cc:1367] T ea4e399a51f24da699e7cd2264de3c0a P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:31.423238 10332 raft_consensus.cc:348] T b08947e5ddd04e7e8dddfa59ca65f713 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:31.423377 10332 raft_consensus.cc:374] T b08947e5ddd04e7e8dddfa59ca65f713 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:31.423404 10332 raft_consensus.cc:729] T b08947e5ddd04e7e8dddfa59ca65f713 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:31.423454 10324 raft_consensus.cc:348] T ea4e399a51f24da699e7cd2264de3c0a P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:31.423516 10332 consensus_queue.cc:260] T b08947e5ddd04e7e8dddfa59ca65f713 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:31.423553 10324 raft_consensus.cc:374] T ea4e399a51f24da699e7cd2264de3c0a P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:31.423584 10324 raft_consensus.cc:729] T ea4e399a51f24da699e7cd2264de3c0a P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:31.423694 10324 consensus_queue.cc:260] T ea4e399a51f24da699e7cd2264de3c0a P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:31.423777 10332 ts_tablet_manager.cc:1398] T b08947e5ddd04e7e8dddfa59ca65f713 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:31.423954 10332 tablet_bootstrap.cc:492] T ea4e399a51f24da699e7cd2264de3c0a P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:31.425181 10332 tablet_bootstrap.cc:654] T ea4e399a51f24da699e7cd2264de3c0a P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:31.426316 10332 tablet_bootstrap.cc:492] T ea4e399a51f24da699e7cd2264de3c0a P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:31.426455 10332 ts_tablet_manager.cc:1367] T ea4e399a51f24da699e7cd2264de3c0a P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:31.426733 10369 tablet_bootstrap.cc:492] T b08947e5ddd04e7e8dddfa59ca65f713 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:31.426877 10372 tablet_bootstrap.cc:492] T 292ed55b431847f39d832ebe61e6c5e4 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:31.427035 10372 ts_tablet_manager.cc:1367] T 292ed55b431847f39d832ebe61e6c5e4 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:59:31.426968 10332 raft_consensus.cc:348] T ea4e399a51f24da699e7cd2264de3c0a P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:31.427112 10332 raft_consensus.cc:374] T ea4e399a51f24da699e7cd2264de3c0a P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:31.427147 10332 raft_consensus.cc:729] T ea4e399a51f24da699e7cd2264de3c0a P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:31.427419 10332 consensus_queue.cc:260] T ea4e399a51f24da699e7cd2264de3c0a P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:31.427793 10372 raft_consensus.cc:348] T 292ed55b431847f39d832ebe61e6c5e4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:31.427887 10369 tablet_bootstrap.cc:654] T b08947e5ddd04e7e8dddfa59ca65f713 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:31.427961 10372 raft_consensus.cc:374] T 292ed55b431847f39d832ebe61e6c5e4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:31.428035 10372 raft_consensus.cc:729] T 292ed55b431847f39d832ebe61e6c5e4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:31.428201 10372 consensus_queue.cc:260] T 292ed55b431847f39d832ebe61e6c5e4 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:31.428730 10324 ts_tablet_manager.cc:1398] T ea4e399a51f24da699e7cd2264de3c0a P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 13:59:31.428917 10324 tablet_bootstrap.cc:492] T 292ed55b431847f39d832ebe61e6c5e4 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:31.429116 10372 ts_tablet_manager.cc:1398] T 292ed55b431847f39d832ebe61e6c5e4 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:31.429946 10324 tablet_bootstrap.cc:654] T 292ed55b431847f39d832ebe61e6c5e4 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:31.430524 10332 ts_tablet_manager.cc:1398] T ea4e399a51f24da699e7cd2264de3c0a P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:59:31.431439 10369 tablet_bootstrap.cc:492] T b08947e5ddd04e7e8dddfa59ca65f713 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:31.431577 10369 ts_tablet_manager.cc:1367] T b08947e5ddd04e7e8dddfa59ca65f713 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 13:59:31.432072 10369 raft_consensus.cc:348] T b08947e5ddd04e7e8dddfa59ca65f713 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:31.432200 10369 raft_consensus.cc:374] T b08947e5ddd04e7e8dddfa59ca65f713 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:31.432250 10369 raft_consensus.cc:729] T b08947e5ddd04e7e8dddfa59ca65f713 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:31.432410 10369 consensus_queue.cc:260] T b08947e5ddd04e7e8dddfa59ca65f713 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:31.432756 10369 ts_tablet_manager.cc:1398] T b08947e5ddd04e7e8dddfa59ca65f713 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:31.433605 10375 tablet_bootstrap.cc:492] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:31.435119 10375 tablet_bootstrap.cc:654] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:31.435818 10370 tablet_bootstrap.cc:492] T ea4e399a51f24da699e7cd2264de3c0a P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:31.436774 10370 tablet_bootstrap.cc:654] T ea4e399a51f24da699e7cd2264de3c0a P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:31.437403 10375 tablet_bootstrap.cc:492] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:31.437498 10375 ts_tablet_manager.cc:1367] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:31.437927 10324 tablet_bootstrap.cc:492] T 292ed55b431847f39d832ebe61e6c5e4 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:31.438028 10324 ts_tablet_manager.cc:1367] T 292ed55b431847f39d832ebe61e6c5e4 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:59:31.438122 10375 raft_consensus.cc:348] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:31.438234 10375 raft_consensus.cc:374] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:31.438297 10375 raft_consensus.cc:729] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:31.438436 10375 consensus_queue.cc:260] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:31.438508 10324 raft_consensus.cc:348] T 292ed55b431847f39d832ebe61e6c5e4 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:31.438632 10324 raft_consensus.cc:374] T 292ed55b431847f39d832ebe61e6c5e4 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:31.438540 10370 tablet_bootstrap.cc:492] T ea4e399a51f24da699e7cd2264de3c0a P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:31.438669 10324 raft_consensus.cc:729] T 292ed55b431847f39d832ebe61e6c5e4 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:31.438768 10370 ts_tablet_manager.cc:1367] T ea4e399a51f24da699e7cd2264de3c0a P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:31.438791 10324 consensus_queue.cc:260] T 292ed55b431847f39d832ebe61e6c5e4 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:31.438877 10375 ts_tablet_manager.cc:1398] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:31.439029 10375 tablet_bootstrap.cc:492] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:31.439047 10324 ts_tablet_manager.cc:1398] T 292ed55b431847f39d832ebe61e6c5e4 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:31.440165 10375 tablet_bootstrap.cc:654] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:31.440819 10370 raft_consensus.cc:348] T ea4e399a51f24da699e7cd2264de3c0a P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:31.440959 10370 raft_consensus.cc:374] T ea4e399a51f24da699e7cd2264de3c0a P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:31.441021 10370 raft_consensus.cc:729] T ea4e399a51f24da699e7cd2264de3c0a P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:31.441159 10370 consensus_queue.cc:260] T ea4e399a51f24da699e7cd2264de3c0a P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:31.441799 10370 ts_tablet_manager.cc:1398] T ea4e399a51f24da699e7cd2264de3c0a P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:31.443228 10375 tablet_bootstrap.cc:492] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:31.443356 10375 ts_tablet_manager.cc:1367] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:59:31.443805 10375 raft_consensus.cc:348] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:31.444034 10375 raft_consensus.cc:374] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:31.444100 10375 raft_consensus.cc:729] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:31.444268 10375 consensus_queue.cc:260] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:31.444532 10375 ts_tablet_manager.cc:1398] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:31.451079 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:31.454558 9211 raft_consensus.cc:1261] T 4bfa080bdf784197814aa52b25439638 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:31.454993 10350 consensus_queue.cc:1035] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:31.455225 6528 raft_consensus.cc:1261] T 4bfa080bdf784197814aa52b25439638 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:31.455622 10350 consensus_queue.cc:1035] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:31.457475 6380 catalog_manager.cc:5576] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091) does not have the latest schema for tablet 4bfa080bdf784197814aa52b25439638 (table rebalancer_dynamic_table [id=b8fddf3bb54a4b46a29445a5a018c8e4]). Expected version 1 got 0
I20240503 13:59:31.458123 6379 catalog_manager.cc:5576] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021) does not have the latest schema for tablet 4bfa080bdf784197814aa52b25439638 (table rebalancer_dynamic_table [id=b8fddf3bb54a4b46a29445a5a018c8e4]). Expected version 1 got 0
I20240503 13:59:31.461558 10385 tablet.cc:1722] T 4bfa080bdf784197814aa52b25439638 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:31.461673 6380 catalog_manager.cc:5576] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091) does not have the latest schema for tablet 4bfa080bdf784197814aa52b25439638 (table rebalancer_dynamic_table [id=b8fddf3bb54a4b46a29445a5a018c8e4]). Expected version 1 got 0
I20240503 13:59:31.461875 10385 tablet.cc:1620] T 4bfa080bdf784197814aa52b25439638 P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:31.462242 10386 tablet.cc:1722] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:31.463186 10341 tablet.cc:1722] T 4bfa080bdf784197814aa52b25439638 P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:31.463429 6379 catalog_manager.cc:5576] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481) does not have the latest schema for tablet 4bfa080bdf784197814aa52b25439638 (table rebalancer_dynamic_table [id=b8fddf3bb54a4b46a29445a5a018c8e4]). Expected version 1 got 0
I20240503 13:59:31.463522 10341 tablet.cc:1620] T 4bfa080bdf784197814aa52b25439638 P 5f46e19644fd4f4b825c86ba6af768da: MemRowSet was empty: no flush needed.
I20240503 13:59:31.465565 10386 tablet.cc:1620] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:31.467761 6386 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=b8fddf3bb54a4b46a29445a5a018c8e4] alter complete (version 1)
I20240503 13:59:31.536386 10273 raft_consensus.cc:482] T b08947e5ddd04e7e8dddfa59ca65f713 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:31.536511 10273 raft_consensus.cc:504] T b08947e5ddd04e7e8dddfa59ca65f713 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:31.536983 10273 leader_election.cc:288] T b08947e5ddd04e7e8dddfa59ca65f713 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:31.537390 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b08947e5ddd04e7e8dddfa59ca65f713" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:31.537400 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b08947e5ddd04e7e8dddfa59ca65f713" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:31.537544 9211 raft_consensus.cc:2454] T b08947e5ddd04e7e8dddfa59ca65f713 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:31.537565 6920 raft_consensus.cc:2454] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:31.537847 6726 leader_election.cc:302] T b08947e5ddd04e7e8dddfa59ca65f713 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:31.537990 10273 raft_consensus.cc:2781] T b08947e5ddd04e7e8dddfa59ca65f713 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:31.538048 10273 raft_consensus.cc:482] T b08947e5ddd04e7e8dddfa59ca65f713 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:31.538074 10273 raft_consensus.cc:3037] T b08947e5ddd04e7e8dddfa59ca65f713 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:31.539022 10273 raft_consensus.cc:504] T b08947e5ddd04e7e8dddfa59ca65f713 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:31.539366 10273 leader_election.cc:288] T b08947e5ddd04e7e8dddfa59ca65f713 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:31.539753 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b08947e5ddd04e7e8dddfa59ca65f713" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:31.539796 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b08947e5ddd04e7e8dddfa59ca65f713" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:31.539888 9211 raft_consensus.cc:3037] T b08947e5ddd04e7e8dddfa59ca65f713 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:31.539901 6920 raft_consensus.cc:3037] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:31.540953 9211 raft_consensus.cc:2454] T b08947e5ddd04e7e8dddfa59ca65f713 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:31.541069 6920 raft_consensus.cc:2454] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:31.541353 6726 leader_election.cc:302] T b08947e5ddd04e7e8dddfa59ca65f713 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:31.541492 10273 raft_consensus.cc:2781] T b08947e5ddd04e7e8dddfa59ca65f713 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:31.541584 10273 raft_consensus.cc:686] T b08947e5ddd04e7e8dddfa59ca65f713 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:31.541730 10273 consensus_queue.cc:237] T b08947e5ddd04e7e8dddfa59ca65f713 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:31.543604 6379 catalog_manager.cc:5484] T b08947e5ddd04e7e8dddfa59ca65f713 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:31.572100 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:31.577047 10386 tablet.cc:1722] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:31.577399 10386 tablet.cc:1620] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:31.577497 10341 tablet.cc:1722] T 4bfa080bdf784197814aa52b25439638 P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:31.577683 10385 tablet.cc:1722] T 4bfa080bdf784197814aa52b25439638 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:31.577826 10341 tablet.cc:1620] T 4bfa080bdf784197814aa52b25439638 P 5f46e19644fd4f4b825c86ba6af768da: MemRowSet was empty: no flush needed.
I20240503 13:59:31.577972 10385 tablet.cc:1620] T 4bfa080bdf784197814aa52b25439638 P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:31.583925 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=b8fddf3bb54a4b46a29445a5a018c8e4] alter complete (version 2)
I20240503 13:59:31.646265 10249 raft_consensus.cc:482] T 292ed55b431847f39d832ebe61e6c5e4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:31.646382 10249 raft_consensus.cc:504] T 292ed55b431847f39d832ebe61e6c5e4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:31.646793 10249 leader_election.cc:288] T 292ed55b431847f39d832ebe61e6c5e4 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:31.647184 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ed55b431847f39d832ebe61e6c5e4" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:31.647212 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ed55b431847f39d832ebe61e6c5e4" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:31.647351 9080 raft_consensus.cc:2454] T 292ed55b431847f39d832ebe61e6c5e4 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:31.647382 6920 raft_consensus.cc:2454] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:31.647670 9278 leader_election.cc:302] T 292ed55b431847f39d832ebe61e6c5e4 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:31.647861 10249 raft_consensus.cc:2781] T 292ed55b431847f39d832ebe61e6c5e4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:31.647922 10249 raft_consensus.cc:482] T 292ed55b431847f39d832ebe61e6c5e4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:31.647949 10249 raft_consensus.cc:3037] T 292ed55b431847f39d832ebe61e6c5e4 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:31.648749 10249 raft_consensus.cc:504] T 292ed55b431847f39d832ebe61e6c5e4 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:31.649055 10249 leader_election.cc:288] T 292ed55b431847f39d832ebe61e6c5e4 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:31.649392 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ed55b431847f39d832ebe61e6c5e4" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:31.649441 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ed55b431847f39d832ebe61e6c5e4" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:31.649518 6920 raft_consensus.cc:3037] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:31.649546 9080 raft_consensus.cc:3037] T 292ed55b431847f39d832ebe61e6c5e4 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:31.650357 9080 raft_consensus.cc:2454] T 292ed55b431847f39d832ebe61e6c5e4 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:31.650405 6920 raft_consensus.cc:2454] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:31.650755 9278 leader_election.cc:302] T 292ed55b431847f39d832ebe61e6c5e4 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:31.650895 10249 raft_consensus.cc:2781] T 292ed55b431847f39d832ebe61e6c5e4 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:31.650974 10249 raft_consensus.cc:686] T 292ed55b431847f39d832ebe61e6c5e4 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:31.651119 10249 consensus_queue.cc:237] T 292ed55b431847f39d832ebe61e6c5e4 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:31.652916 6379 catalog_manager.cc:5484] T 292ed55b431847f39d832ebe61e6c5e4 P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:31.686273 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:31.691890 10386 tablet.cc:1722] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:31.692314 10385 tablet.cc:1722] T 4bfa080bdf784197814aa52b25439638 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:31.692355 10341 tablet.cc:1722] T 4bfa080bdf784197814aa52b25439638 P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:31.697276 10387 raft_consensus.cc:482] T ea4e399a51f24da699e7cd2264de3c0a P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:31.697368 10387 raft_consensus.cc:504] T ea4e399a51f24da699e7cd2264de3c0a P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:31.697260 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=b8fddf3bb54a4b46a29445a5a018c8e4] alter complete (version 3)
I20240503 13:59:31.698076 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea4e399a51f24da699e7cd2264de3c0a" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:31.698195 9080 raft_consensus.cc:2454] T ea4e399a51f24da699e7cd2264de3c0a P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:31.698523 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea4e399a51f24da699e7cd2264de3c0a" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:31.698675 6658 raft_consensus.cc:2454] T ea4e399a51f24da699e7cd2264de3c0a P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:31.697778 10387 leader_election.cc:288] T ea4e399a51f24da699e7cd2264de3c0a P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:31.698487 9147 leader_election.cc:302] T ea4e399a51f24da699e7cd2264de3c0a P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:31.699371 10387 raft_consensus.cc:2781] T ea4e399a51f24da699e7cd2264de3c0a P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:31.699427 10387 raft_consensus.cc:482] T ea4e399a51f24da699e7cd2264de3c0a P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:31.699453 10387 raft_consensus.cc:3037] T ea4e399a51f24da699e7cd2264de3c0a P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:31.700625 10387 raft_consensus.cc:504] T ea4e399a51f24da699e7cd2264de3c0a P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:31.700995 10387 leader_election.cc:288] T ea4e399a51f24da699e7cd2264de3c0a P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:31.701246 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea4e399a51f24da699e7cd2264de3c0a" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:31.701362 9080 raft_consensus.cc:3037] T ea4e399a51f24da699e7cd2264de3c0a P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:31.701570 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea4e399a51f24da699e7cd2264de3c0a" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:31.701678 6658 raft_consensus.cc:3037] T ea4e399a51f24da699e7cd2264de3c0a P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:31.702241 9080 raft_consensus.cc:2454] T ea4e399a51f24da699e7cd2264de3c0a P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:31.702517 9147 leader_election.cc:302] T ea4e399a51f24da699e7cd2264de3c0a P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:31.702672 6658 raft_consensus.cc:2454] T ea4e399a51f24da699e7cd2264de3c0a P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:31.702721 10387 raft_consensus.cc:2781] T ea4e399a51f24da699e7cd2264de3c0a P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:31.702811 10387 raft_consensus.cc:686] T ea4e399a51f24da699e7cd2264de3c0a P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 13:59:31.702934 10387 consensus_queue.cc:237] T ea4e399a51f24da699e7cd2264de3c0a P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:31.704643 6386 catalog_manager.cc:5484] T ea4e399a51f24da699e7cd2264de3c0a P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:31.798871 6386 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:31.799027 6386 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:31.801080 6386 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 4bfa080bdf784197814aa52b25439638
I20240503 13:59:31.801682 6508 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bfa080bdf784197814aa52b25439638 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:31 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:31.801744 9190 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bfa080bdf784197814aa52b25439638 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:31 UTC) from {username='slave'} at 127.0.0.1:57960
I20240503 13:59:31.802011 9058 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bfa080bdf784197814aa52b25439638 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:31 UTC) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:31.802073 10392 tablet_replica.cc:330] T 4bfa080bdf784197814aa52b25439638 P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 13:59:31.802177 10391 tablet_replica.cc:330] T 4bfa080bdf784197814aa52b25439638 P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:31.802212 10392 raft_consensus.cc:2229] T 4bfa080bdf784197814aa52b25439638 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:31.802359 10392 raft_consensus.cc:2258] T 4bfa080bdf784197814aa52b25439638 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:31.802366 10393 tablet_replica.cc:330] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:31.802371 10391 raft_consensus.cc:2229] T 4bfa080bdf784197814aa52b25439638 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:31.802489 10391 raft_consensus.cc:2258] T 4bfa080bdf784197814aa52b25439638 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:31.802492 10393 raft_consensus.cc:2229] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:31.802701 10393 raft_consensus.cc:2258] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:31.803118 10392 ts_tablet_manager.cc:1875] T 4bfa080bdf784197814aa52b25439638 P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:31.803141 10391 ts_tablet_manager.cc:1875] T 4bfa080bdf784197814aa52b25439638 P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:31.803318 10393 ts_tablet_manager.cc:1875] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:31.805286 10392 ts_tablet_manager.cc:1888] T 4bfa080bdf784197814aa52b25439638 P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:31.805400 10392 log.cc:1196] T 4bfa080bdf784197814aa52b25439638 P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/4bfa080bdf784197814aa52b25439638
I20240503 13:59:31.805449 10391 ts_tablet_manager.cc:1888] T 4bfa080bdf784197814aa52b25439638 P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:31.805549 10391 log.cc:1196] T 4bfa080bdf784197814aa52b25439638 P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/4bfa080bdf784197814aa52b25439638
I20240503 13:59:31.805720 10392 ts_tablet_manager.cc:1909] T 4bfa080bdf784197814aa52b25439638 P ab1d21935d6049888eb0ceccc3c9d735: Deleting consensus metadata
I20240503 13:59:31.805881 10391 ts_tablet_manager.cc:1909] T 4bfa080bdf784197814aa52b25439638 P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:31.806473 10393 ts_tablet_manager.cc:1888] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:31.806547 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet 4bfa080bdf784197814aa52b25439638 (table rebalancer_dynamic_table_new_name [id=b8fddf3bb54a4b46a29445a5a018c8e4]) successfully deleted
I20240503 13:59:31.806689 10393 log.cc:1196] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/4bfa080bdf784197814aa52b25439638
I20240503 13:59:31.806747 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 4bfa080bdf784197814aa52b25439638 (table rebalancer_dynamic_table_new_name [id=b8fddf3bb54a4b46a29445a5a018c8e4]) successfully deleted
I20240503 13:59:31.807730 10393 ts_tablet_manager.cc:1909] T 4bfa080bdf784197814aa52b25439638 P 8d574fb4a89d4eea947c859470d44dd4: Deleting consensus metadata
I20240503 13:59:31.809199 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet 4bfa080bdf784197814aa52b25439638 (table rebalancer_dynamic_table_new_name [id=b8fddf3bb54a4b46a29445a5a018c8e4]) successfully deleted
I20240503 13:59:31.902833 6386 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:31.908231 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 823901cb1e5d4a32a63dc9731be11a14 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d6b1dd97a96d4bdb99f410a1a19246d8]), partition=
I20240503 13:59:31.908524 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 823901cb1e5d4a32a63dc9731be11a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:31.908522 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 823901cb1e5d4a32a63dc9731be11a14 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d6b1dd97a96d4bdb99f410a1a19246d8]), partition=
I20240503 13:59:31.908764 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 823901cb1e5d4a32a63dc9731be11a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:31.910521 10396 tablet_bootstrap.cc:492] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:31.910881 9323 tablet_service.cc:1467] Processing CreateTablet for tablet 823901cb1e5d4a32a63dc9731be11a14 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d6b1dd97a96d4bdb99f410a1a19246d8]), partition=
I20240503 13:59:31.911075 10397 tablet_bootstrap.cc:492] T 823901cb1e5d4a32a63dc9731be11a14 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:31.911126 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 823901cb1e5d4a32a63dc9731be11a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:31.911921 10396 tablet_bootstrap.cc:654] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:31.912699 10397 tablet_bootstrap.cc:654] T 823901cb1e5d4a32a63dc9731be11a14 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:31.913368 10372 tablet_bootstrap.cc:492] T 823901cb1e5d4a32a63dc9731be11a14 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:31.914280 10372 tablet_bootstrap.cc:654] T 823901cb1e5d4a32a63dc9731be11a14 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:31.914487 10397 tablet_bootstrap.cc:492] T 823901cb1e5d4a32a63dc9731be11a14 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:31.914582 10397 ts_tablet_manager.cc:1367] T 823901cb1e5d4a32a63dc9731be11a14 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:31.915089 10397 raft_consensus.cc:348] T 823901cb1e5d4a32a63dc9731be11a14 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:31.915201 10397 raft_consensus.cc:374] T 823901cb1e5d4a32a63dc9731be11a14 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:31.915248 10397 raft_consensus.cc:729] T 823901cb1e5d4a32a63dc9731be11a14 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:31.915388 10397 consensus_queue.cc:260] T 823901cb1e5d4a32a63dc9731be11a14 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:31.915699 10397 ts_tablet_manager.cc:1398] T 823901cb1e5d4a32a63dc9731be11a14 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:31.915740 10372 tablet_bootstrap.cc:492] T 823901cb1e5d4a32a63dc9731be11a14 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:31.915812 10372 ts_tablet_manager.cc:1367] T 823901cb1e5d4a32a63dc9731be11a14 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:31.916275 10396 tablet_bootstrap.cc:492] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:31.916247 10372 raft_consensus.cc:348] T 823901cb1e5d4a32a63dc9731be11a14 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:31.916357 10372 raft_consensus.cc:374] T 823901cb1e5d4a32a63dc9731be11a14 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:31.916380 10372 raft_consensus.cc:729] T 823901cb1e5d4a32a63dc9731be11a14 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:31.916371 10396 ts_tablet_manager.cc:1367] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:59:31.916496 10372 consensus_queue.cc:260] T 823901cb1e5d4a32a63dc9731be11a14 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:31.916719 10372 ts_tablet_manager.cc:1398] T 823901cb1e5d4a32a63dc9731be11a14 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:31.916800 10396 raft_consensus.cc:348] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:31.916923 10396 raft_consensus.cc:374] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:31.916954 10396 raft_consensus.cc:729] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:31.917106 10396 consensus_queue.cc:260] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:31.917400 10396 ts_tablet_manager.cc:1398] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:32.050897 10254 raft_consensus.cc:482] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:32.051033 10254 raft_consensus.cc:504] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:32.051434 10254 leader_election.cc:288] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:32.051784 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "823901cb1e5d4a32a63dc9731be11a14" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:32.051836 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "823901cb1e5d4a32a63dc9731be11a14" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:32.051954 8947 raft_consensus.cc:2454] T 823901cb1e5d4a32a63dc9731be11a14 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:32.051954 9343 raft_consensus.cc:2454] T 823901cb1e5d4a32a63dc9731be11a14 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:32.052296 6465 leader_election.cc:302] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:32.052489 10254 raft_consensus.cc:2781] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:32.052553 10254 raft_consensus.cc:482] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:32.052608 10254 raft_consensus.cc:3037] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.053687 10254 raft_consensus.cc:504] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:32.054020 10254 leader_election.cc:288] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:32.054330 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "823901cb1e5d4a32a63dc9731be11a14" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:32.054366 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "823901cb1e5d4a32a63dc9731be11a14" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:32.054451 9343 raft_consensus.cc:3037] T 823901cb1e5d4a32a63dc9731be11a14 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.054474 8947 raft_consensus.cc:3037] T 823901cb1e5d4a32a63dc9731be11a14 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.055171 9343 raft_consensus.cc:2454] T 823901cb1e5d4a32a63dc9731be11a14 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:32.055382 8947 raft_consensus.cc:2454] T 823901cb1e5d4a32a63dc9731be11a14 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:32.056385 6465 leader_election.cc:302] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:32.056566 10254 raft_consensus.cc:2781] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:32.056728 10254 raft_consensus.cc:686] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Becoming Leader. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Running, Role: LEADER
I20240503 13:59:32.056941 10254 consensus_queue.cc:237] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:32.058812 6386 catalog_manager.cc:5484] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:32.071413 10273 consensus_queue.cc:1035] T b08947e5ddd04e7e8dddfa59ca65f713 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:32.074671 10394 consensus_queue.cc:1035] T b08947e5ddd04e7e8dddfa59ca65f713 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:32.111020 10339 consensus_queue.cc:1035] T 292ed55b431847f39d832ebe61e6c5e4 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:32.114645 10298 consensus_queue.cc:1035] T 292ed55b431847f39d832ebe61e6c5e4 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:32.146521 10387 consensus_queue.cc:1035] T ea4e399a51f24da699e7cd2264de3c0a P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:32.149865 10304 consensus_queue.cc:1035] T ea4e399a51f24da699e7cd2264de3c0a P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:32.197974 6386 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:32.202433 9343 raft_consensus.cc:1261] T 823901cb1e5d4a32a63dc9731be11a14 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:32.202615 8947 raft_consensus.cc:1261] T 823901cb1e5d4a32a63dc9731be11a14 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:32.203027 10254 consensus_queue.cc:1035] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:32.203663 10402 consensus_queue.cc:1035] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:32.206022 10418 tablet.cc:1722] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:32.206233 6386 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet 823901cb1e5d4a32a63dc9731be11a14 (table rebalancer_dynamic_table [id=d6b1dd97a96d4bdb99f410a1a19246d8]). Expected version 1 got 0
I20240503 13:59:32.206951 10418 tablet.cc:1620] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da: MemRowSet was empty: no flush needed.
I20240503 13:59:32.207193 6379 catalog_manager.cc:5576] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931) does not have the latest schema for tablet 823901cb1e5d4a32a63dc9731be11a14 (table rebalancer_dynamic_table [id=d6b1dd97a96d4bdb99f410a1a19246d8]). Expected version 1 got 0
I20240503 13:59:32.207310 10419 tablet.cc:1722] T 823901cb1e5d4a32a63dc9731be11a14 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:32.207616 10419 tablet.cc:1620] T 823901cb1e5d4a32a63dc9731be11a14 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:32.209434 6386 catalog_manager.cc:5576] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021) does not have the latest schema for tablet 823901cb1e5d4a32a63dc9731be11a14 (table rebalancer_dynamic_table [id=d6b1dd97a96d4bdb99f410a1a19246d8]). Expected version 1 got 0
I20240503 13:59:32.209784 6383 catalog_manager.cc:5576] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931) does not have the latest schema for tablet 823901cb1e5d4a32a63dc9731be11a14 (table rebalancer_dynamic_table [id=d6b1dd97a96d4bdb99f410a1a19246d8]). Expected version 1 got 0
I20240503 13:59:32.212062 10421 tablet.cc:1722] T 823901cb1e5d4a32a63dc9731be11a14 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:32.212227 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=d6b1dd97a96d4bdb99f410a1a19246d8] alter complete (version 1)
I20240503 13:59:32.212366 10421 tablet.cc:1620] T 823901cb1e5d4a32a63dc9731be11a14 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:32.230218 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:32.236743 6902 tablet_service.cc:1467] Processing CreateTablet for tablet 6ee85f4483004c1f9499203c7323706d (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=69fc27437d404cf4a64ea4f3de479bfc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.236743 9323 tablet_service.cc:1467] Processing CreateTablet for tablet 6ee85f4483004c1f9499203c7323706d (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=69fc27437d404cf4a64ea4f3de479bfc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.237071 6902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ee85f4483004c1f9499203c7323706d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.237268 9058 tablet_service.cc:1467] Processing CreateTablet for tablet fd45a0cf3a2d4657b02016bc60c5be34 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=69fc27437d404cf4a64ea4f3de479bfc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.237434 9190 tablet_service.cc:1467] Processing CreateTablet for tablet 6ee85f4483004c1f9499203c7323706d (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=69fc27437d404cf4a64ea4f3de479bfc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.237532 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd45a0cf3a2d4657b02016bc60c5be34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.237735 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ee85f4483004c1f9499203c7323706d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.239564 9061 tablet_service.cc:1467] Processing CreateTablet for tablet 63d2b0a8c3734720ba7d72a36d344cba (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=69fc27437d404cf4a64ea4f3de479bfc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.239751 10422 tablet_bootstrap.cc:492] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:32.239691 8928 tablet_service.cc:1467] Processing CreateTablet for tablet fd45a0cf3a2d4657b02016bc60c5be34 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=69fc27437d404cf4a64ea4f3de479bfc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.239812 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63d2b0a8c3734720ba7d72a36d344cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.239998 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd45a0cf3a2d4657b02016bc60c5be34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.240115 10423 tablet_bootstrap.cc:492] T 6ee85f4483004c1f9499203c7323706d P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:32.240864 10422 tablet_bootstrap.cc:654] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.241114 10423 tablet_bootstrap.cc:654] T 6ee85f4483004c1f9499203c7323706d P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.241839 10426 tablet_bootstrap.cc:492] T 63d2b0a8c3734720ba7d72a36d344cba P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:32.242379 10397 tablet_bootstrap.cc:492] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:32.242662 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ee85f4483004c1f9499203c7323706d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.242707 9190 tablet_service.cc:1467] Processing CreateTablet for tablet 63d2b0a8c3734720ba7d72a36d344cba (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=69fc27437d404cf4a64ea4f3de479bfc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.242923 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63d2b0a8c3734720ba7d72a36d344cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.243331 10397 tablet_bootstrap.cc:654] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.243376 10423 tablet_bootstrap.cc:492] T 6ee85f4483004c1f9499203c7323706d P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:32.243448 10423 ts_tablet_manager.cc:1367] T 6ee85f4483004c1f9499203c7323706d P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:32.243892 10423 raft_consensus.cc:348] T 6ee85f4483004c1f9499203c7323706d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:32.243993 10423 raft_consensus.cc:374] T 6ee85f4483004c1f9499203c7323706d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.244029 10423 raft_consensus.cc:729] T 6ee85f4483004c1f9499203c7323706d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.244158 10423 consensus_queue.cc:260] T 6ee85f4483004c1f9499203c7323706d P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:32.245993 10372 tablet_bootstrap.cc:492] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:32.246138 10423 ts_tablet_manager.cc:1398] T 6ee85f4483004c1f9499203c7323706d P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:32.246290 10423 tablet_bootstrap.cc:492] T 63d2b0a8c3734720ba7d72a36d344cba P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:32.246997 10372 tablet_bootstrap.cc:654] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.247494 10423 tablet_bootstrap.cc:654] T 63d2b0a8c3734720ba7d72a36d344cba P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.248472 10426 tablet_bootstrap.cc:654] T 63d2b0a8c3734720ba7d72a36d344cba P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.248517 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 63d2b0a8c3734720ba7d72a36d344cba (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=69fc27437d404cf4a64ea4f3de479bfc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.248800 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63d2b0a8c3734720ba7d72a36d344cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.249326 10423 tablet_bootstrap.cc:492] T 63d2b0a8c3734720ba7d72a36d344cba P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:32.249397 10423 ts_tablet_manager.cc:1367] T 63d2b0a8c3734720ba7d72a36d344cba P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:32.249585 10426 tablet_bootstrap.cc:492] T 63d2b0a8c3734720ba7d72a36d344cba P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:32.249585 10372 tablet_bootstrap.cc:492] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:32.249662 10426 ts_tablet_manager.cc:1367] T 63d2b0a8c3734720ba7d72a36d344cba P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:59:32.249679 10372 ts_tablet_manager.cc:1367] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240503 13:59:32.249969 10423 raft_consensus.cc:348] T 63d2b0a8c3734720ba7d72a36d344cba P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:32.250115 10423 raft_consensus.cc:374] T 63d2b0a8c3734720ba7d72a36d344cba P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.250161 10423 raft_consensus.cc:729] T 63d2b0a8c3734720ba7d72a36d344cba P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.250144 10372 raft_consensus.cc:348] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:32.250238 10372 raft_consensus.cc:374] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.250285 10372 raft_consensus.cc:729] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.250269 10423 consensus_queue.cc:260] T 63d2b0a8c3734720ba7d72a36d344cba P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:32.250277 10426 raft_consensus.cc:348] T 63d2b0a8c3734720ba7d72a36d344cba P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:32.250367 10426 raft_consensus.cc:374] T 63d2b0a8c3734720ba7d72a36d344cba P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.250396 10426 raft_consensus.cc:729] T 63d2b0a8c3734720ba7d72a36d344cba P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.250396 10372 consensus_queue.cc:260] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:32.250520 10426 consensus_queue.cc:260] T 63d2b0a8c3734720ba7d72a36d344cba P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:32.250628 10423 ts_tablet_manager.cc:1398] T 63d2b0a8c3734720ba7d72a36d344cba P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:32.250695 10372 ts_tablet_manager.cc:1398] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:32.250804 10426 ts_tablet_manager.cc:1398] T 63d2b0a8c3734720ba7d72a36d344cba P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:32.251180 10431 tablet_bootstrap.cc:492] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:32.252418 10431 tablet_bootstrap.cc:654] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.252681 10426 tablet_bootstrap.cc:492] T fd45a0cf3a2d4657b02016bc60c5be34 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:32.253559 10426 tablet_bootstrap.cc:654] T fd45a0cf3a2d4657b02016bc60c5be34 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.254345 10431 tablet_bootstrap.cc:492] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:32.254427 10431 ts_tablet_manager.cc:1367] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:32.254551 10426 tablet_bootstrap.cc:492] T fd45a0cf3a2d4657b02016bc60c5be34 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:32.254628 10426 ts_tablet_manager.cc:1367] T fd45a0cf3a2d4657b02016bc60c5be34 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:32.254715 9323 tablet_service.cc:1467] Processing CreateTablet for tablet fd45a0cf3a2d4657b02016bc60c5be34 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=69fc27437d404cf4a64ea4f3de479bfc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.254838 10431 raft_consensus.cc:348] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:32.254933 10431 raft_consensus.cc:374] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.254957 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd45a0cf3a2d4657b02016bc60c5be34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.254964 10431 raft_consensus.cc:729] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.255038 10426 raft_consensus.cc:348] T fd45a0cf3a2d4657b02016bc60c5be34 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:32.255077 10431 consensus_queue.cc:260] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:32.255136 10426 raft_consensus.cc:374] T fd45a0cf3a2d4657b02016bc60c5be34 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.255167 10426 raft_consensus.cc:729] T fd45a0cf3a2d4657b02016bc60c5be34 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.255317 10426 consensus_queue.cc:260] T fd45a0cf3a2d4657b02016bc60c5be34 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:32.255398 10431 ts_tablet_manager.cc:1398] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:32.255630 10426 ts_tablet_manager.cc:1398] T fd45a0cf3a2d4657b02016bc60c5be34 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:32.257180 10372 tablet_bootstrap.cc:492] T fd45a0cf3a2d4657b02016bc60c5be34 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:32.258155 10372 tablet_bootstrap.cc:654] T fd45a0cf3a2d4657b02016bc60c5be34 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.258647 10397 tablet_bootstrap.cc:492] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:32.258734 10397 ts_tablet_manager.cc:1367] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.002s
I20240503 13:59:32.259200 10397 raft_consensus.cc:348] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:32.259341 10397 raft_consensus.cc:374] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.259375 10397 raft_consensus.cc:729] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.259431 10372 tablet_bootstrap.cc:492] T fd45a0cf3a2d4657b02016bc60c5be34 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:32.259500 10372 ts_tablet_manager.cc:1367] T fd45a0cf3a2d4657b02016bc60c5be34 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:32.259568 10397 consensus_queue.cc:260] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:32.259840 10397 ts_tablet_manager.cc:1398] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:32.259958 10372 raft_consensus.cc:348] T fd45a0cf3a2d4657b02016bc60c5be34 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:32.260072 10372 raft_consensus.cc:374] T fd45a0cf3a2d4657b02016bc60c5be34 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.260104 10372 raft_consensus.cc:729] T fd45a0cf3a2d4657b02016bc60c5be34 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.260217 10372 consensus_queue.cc:260] T fd45a0cf3a2d4657b02016bc60c5be34 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:32.260387 10422 tablet_bootstrap.cc:492] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:32.260474 10372 ts_tablet_manager.cc:1398] T fd45a0cf3a2d4657b02016bc60c5be34 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:32.260484 10422 ts_tablet_manager.cc:1367] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240503 13:59:32.260994 10422 raft_consensus.cc:348] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:32.261116 10422 raft_consensus.cc:374] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.261148 10422 raft_consensus.cc:729] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.261288 10422 consensus_queue.cc:260] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:32.261601 10422 ts_tablet_manager.cc:1398] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:32.316390 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:32.321818 10418 tablet.cc:1722] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:32.322175 10418 tablet.cc:1620] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da: MemRowSet was empty: no flush needed.
I20240503 13:59:32.322539 10421 tablet.cc:1722] T 823901cb1e5d4a32a63dc9731be11a14 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:32.322880 10421 tablet.cc:1620] T 823901cb1e5d4a32a63dc9731be11a14 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:32.322609 10298 raft_consensus.cc:482] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:32.323122 10298 raft_consensus.cc:504] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:32.323557 10298 leader_election.cc:288] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:32.324407 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee85f4483004c1f9499203c7323706d" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:32.324571 6920 raft_consensus.cc:2454] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:32.324793 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee85f4483004c1f9499203c7323706d" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:32.324939 9211 raft_consensus.cc:2454] T 6ee85f4483004c1f9499203c7323706d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:32.325242 9278 leader_election.cc:302] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:32.325544 10298 raft_consensus.cc:2781] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:32.325608 10298 raft_consensus.cc:482] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:32.325647 10298 raft_consensus.cc:3037] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.326435 10298 raft_consensus.cc:504] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:32.326787 10298 leader_election.cc:288] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:32.327124 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee85f4483004c1f9499203c7323706d" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:32.327239 6920 raft_consensus.cc:3037] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.328219 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee85f4483004c1f9499203c7323706d" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:32.328266 6920 raft_consensus.cc:2454] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:32.328338 9211 raft_consensus.cc:3037] T 6ee85f4483004c1f9499203c7323706d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.329227 9211 raft_consensus.cc:2454] T 6ee85f4483004c1f9499203c7323706d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:32.329375 9278 leader_election.cc:302] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:32.329375 10419 tablet.cc:1722] T 823901cb1e5d4a32a63dc9731be11a14 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:32.329679 10419 tablet.cc:1620] T 823901cb1e5d4a32a63dc9731be11a14 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:32.330310 6383 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=d6b1dd97a96d4bdb99f410a1a19246d8] alter complete (version 2)
I20240503 13:59:32.330430 10298 raft_consensus.cc:2781] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:32.330500 10298 raft_consensus.cc:686] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:32.330617 10298 consensus_queue.cc:237] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:32.333282 6379 catalog_manager.cc:5484] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:32.417490 10436 raft_consensus.cc:482] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:32.417626 10436 raft_consensus.cc:504] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:32.418092 10436 leader_election.cc:288] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:32.418489 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd45a0cf3a2d4657b02016bc60c5be34" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:32.418637 9343 raft_consensus.cc:2454] T fd45a0cf3a2d4657b02016bc60c5be34 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:32.418641 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd45a0cf3a2d4657b02016bc60c5be34" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:32.418763 9080 raft_consensus.cc:2454] T fd45a0cf3a2d4657b02016bc60c5be34 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:32.418917 8884 leader_election.cc:302] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 0d77752181e944698e809e325e5937e6; no voters:
I20240503 13:59:32.419117 10436 raft_consensus.cc:2781] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:32.419174 10436 raft_consensus.cc:482] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:32.419225 10436 raft_consensus.cc:3037] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.420106 10436 raft_consensus.cc:504] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:32.420423 10436 leader_election.cc:288] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:32.420688 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd45a0cf3a2d4657b02016bc60c5be34" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:32.420688 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd45a0cf3a2d4657b02016bc60c5be34" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:32.420806 9080 raft_consensus.cc:3037] T fd45a0cf3a2d4657b02016bc60c5be34 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.420806 9343 raft_consensus.cc:3037] T fd45a0cf3a2d4657b02016bc60c5be34 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.421702 9343 raft_consensus.cc:2454] T fd45a0cf3a2d4657b02016bc60c5be34 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:32.421702 9080 raft_consensus.cc:2454] T fd45a0cf3a2d4657b02016bc60c5be34 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:32.422019 8884 leader_election.cc:302] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 0d77752181e944698e809e325e5937e6; no voters:
I20240503 13:59:32.422237 10436 raft_consensus.cc:2781] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:32.422331 10436 raft_consensus.cc:686] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 13:59:32.422461 10436 consensus_queue.cc:237] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:32.424335 6379 catalog_manager.cc:5484] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:32.433775 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:32.438737 10418 tablet.cc:1722] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:32.439118 10419 tablet.cc:1722] T 823901cb1e5d4a32a63dc9731be11a14 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:32.439576 10421 tablet.cc:1722] T 823901cb1e5d4a32a63dc9731be11a14 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:32.443895 6383 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=d6b1dd97a96d4bdb99f410a1a19246d8] alter complete (version 3)
I20240503 13:59:32.457744 10196 raft_consensus.cc:482] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:32.457880 10196 raft_consensus.cc:504] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:32.458266 10196 leader_election.cc:288] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:32.458606 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d2b0a8c3734720ba7d72a36d344cba" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:32.458667 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d2b0a8c3734720ba7d72a36d344cba" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:32.458745 9211 raft_consensus.cc:2454] T 63d2b0a8c3734720ba7d72a36d344cba P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:32.458819 9080 raft_consensus.cc:2454] T 63d2b0a8c3734720ba7d72a36d344cba P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:32.459062 6595 leader_election.cc:302] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:32.459235 10196 raft_consensus.cc:2781] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:32.459313 10196 raft_consensus.cc:482] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:32.459357 10196 raft_consensus.cc:3037] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.460530 10196 raft_consensus.cc:504] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:32.460891 10196 leader_election.cc:288] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:32.461177 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d2b0a8c3734720ba7d72a36d344cba" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:32.461294 9211 raft_consensus.cc:3037] T 63d2b0a8c3734720ba7d72a36d344cba P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.461311 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d2b0a8c3734720ba7d72a36d344cba" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:32.461445 9080 raft_consensus.cc:3037] T 63d2b0a8c3734720ba7d72a36d344cba P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.462229 9211 raft_consensus.cc:2454] T 63d2b0a8c3734720ba7d72a36d344cba P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:32.462294 9080 raft_consensus.cc:2454] T 63d2b0a8c3734720ba7d72a36d344cba P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:32.462715 6595 leader_election.cc:302] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:32.463058 10196 raft_consensus.cc:2781] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:32.463268 10196 raft_consensus.cc:686] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 13:59:32.463434 10196 consensus_queue.cc:237] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:32.465427 6383 catalog_manager.cc:5484] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:32.545593 6383 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:32.545797 6383 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:32.547904 6383 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 823901cb1e5d4a32a63dc9731be11a14
I20240503 13:59:32.548540 8928 tablet_service.cc:1514] Processing DeleteTablet for tablet 823901cb1e5d4a32a63dc9731be11a14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:32 UTC) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:32.548548 9323 tablet_service.cc:1514] Processing DeleteTablet for tablet 823901cb1e5d4a32a63dc9731be11a14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:32 UTC) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:32.548548 6508 tablet_service.cc:1514] Processing DeleteTablet for tablet 823901cb1e5d4a32a63dc9731be11a14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:32 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:32.548889 10440 tablet_replica.cc:330] T 823901cb1e5d4a32a63dc9731be11a14 P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:32.549033 10440 raft_consensus.cc:2229] T 823901cb1e5d4a32a63dc9731be11a14 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:32.549152 10440 raft_consensus.cc:2258] T 823901cb1e5d4a32a63dc9731be11a14 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:32.549389 10441 tablet_replica.cc:330] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:32.549536 10441 raft_consensus.cc:2229] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:32.549827 10440 ts_tablet_manager.cc:1875] T 823901cb1e5d4a32a63dc9731be11a14 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:32.549882 10441 raft_consensus.cc:2258] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:32.550132 10442 tablet_replica.cc:330] T 823901cb1e5d4a32a63dc9731be11a14 P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:32.550287 10442 raft_consensus.cc:2229] T 823901cb1e5d4a32a63dc9731be11a14 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:32.550743 10441 ts_tablet_manager.cc:1875] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:32.552243 10440 ts_tablet_manager.cc:1888] T 823901cb1e5d4a32a63dc9731be11a14 P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:32.552313 10440 log.cc:1196] T 823901cb1e5d4a32a63dc9731be11a14 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/823901cb1e5d4a32a63dc9731be11a14
I20240503 13:59:32.552610 10440 ts_tablet_manager.cc:1909] T 823901cb1e5d4a32a63dc9731be11a14 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting consensus metadata
I20240503 13:59:32.552953 10441 ts_tablet_manager.cc:1888] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:32.553014 10441 log.cc:1196] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/823901cb1e5d4a32a63dc9731be11a14
I20240503 13:59:32.553277 10441 ts_tablet_manager.cc:1909] T 823901cb1e5d4a32a63dc9731be11a14 P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:32.553278 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 823901cb1e5d4a32a63dc9731be11a14 (table rebalancer_dynamic_table_new_name [id=d6b1dd97a96d4bdb99f410a1a19246d8]) successfully deleted
I20240503 13:59:32.553505 10442 raft_consensus.cc:2258] T 823901cb1e5d4a32a63dc9731be11a14 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:32.554101 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 823901cb1e5d4a32a63dc9731be11a14 (table rebalancer_dynamic_table_new_name [id=d6b1dd97a96d4bdb99f410a1a19246d8]) successfully deleted
I20240503 13:59:32.554195 10442 ts_tablet_manager.cc:1875] T 823901cb1e5d4a32a63dc9731be11a14 P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:32.555936 10442 ts_tablet_manager.cc:1888] T 823901cb1e5d4a32a63dc9731be11a14 P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:32.556005 10442 log.cc:1196] T 823901cb1e5d4a32a63dc9731be11a14 P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/823901cb1e5d4a32a63dc9731be11a14
I20240503 13:59:32.556263 10442 ts_tablet_manager.cc:1909] T 823901cb1e5d4a32a63dc9731be11a14 P 0d77752181e944698e809e325e5937e6: Deleting consensus metadata
I20240503 13:59:32.556937 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet 823901cb1e5d4a32a63dc9731be11a14 (table rebalancer_dynamic_table_new_name [id=d6b1dd97a96d4bdb99f410a1a19246d8]) successfully deleted
I20240503 13:59:32.649910 6386 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:32.655475 6769 tablet_service.cc:1467] Processing CreateTablet for tablet 9de084b227e34d81a4501025e53beded (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bed9b6a9c9bf4c0fbd30c2988b7fc824]), partition=
I20240503 13:59:32.655758 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9de084b227e34d81a4501025e53beded. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.658288 10445 tablet_bootstrap.cc:492] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:32.658861 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 9de084b227e34d81a4501025e53beded (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bed9b6a9c9bf4c0fbd30c2988b7fc824]), partition=
I20240503 13:59:32.658902 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 9de084b227e34d81a4501025e53beded (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bed9b6a9c9bf4c0fbd30c2988b7fc824]), partition=
I20240503 13:59:32.659113 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9de084b227e34d81a4501025e53beded. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.659130 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9de084b227e34d81a4501025e53beded. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.659507 10445 tablet_bootstrap.cc:654] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.660637 10445 tablet_bootstrap.cc:492] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:32.660727 10445 ts_tablet_manager.cc:1367] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:32.661118 10426 tablet_bootstrap.cc:492] T 9de084b227e34d81a4501025e53beded P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:32.661418 10397 tablet_bootstrap.cc:492] T 9de084b227e34d81a4501025e53beded P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:32.661252 10445 raft_consensus.cc:348] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:32.661476 10445 raft_consensus.cc:374] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.661525 10445 raft_consensus.cc:729] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.661641 10445 consensus_queue.cc:260] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:32.661978 10426 tablet_bootstrap.cc:654] T 9de084b227e34d81a4501025e53beded P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.662348 10397 tablet_bootstrap.cc:654] T 9de084b227e34d81a4501025e53beded P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.663365 10426 tablet_bootstrap.cc:492] T 9de084b227e34d81a4501025e53beded P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:32.663372 10445 ts_tablet_manager.cc:1398] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:32.663462 10426 ts_tablet_manager.cc:1367] T 9de084b227e34d81a4501025e53beded P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:32.663890 10426 raft_consensus.cc:348] T 9de084b227e34d81a4501025e53beded P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:32.663988 10426 raft_consensus.cc:374] T 9de084b227e34d81a4501025e53beded P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.664024 10426 raft_consensus.cc:729] T 9de084b227e34d81a4501025e53beded P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.664142 10426 consensus_queue.cc:260] T 9de084b227e34d81a4501025e53beded P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:32.664444 10426 ts_tablet_manager.cc:1398] T 9de084b227e34d81a4501025e53beded P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:32.664620 10397 tablet_bootstrap.cc:492] T 9de084b227e34d81a4501025e53beded P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:32.664698 10397 ts_tablet_manager.cc:1367] T 9de084b227e34d81a4501025e53beded P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:32.665133 10397 raft_consensus.cc:348] T 9de084b227e34d81a4501025e53beded P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:32.665206 10397 raft_consensus.cc:374] T 9de084b227e34d81a4501025e53beded P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.665225 10397 raft_consensus.cc:729] T 9de084b227e34d81a4501025e53beded P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.665315 10397 consensus_queue.cc:260] T 9de084b227e34d81a4501025e53beded P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:32.665537 10397 ts_tablet_manager.cc:1398] T 9de084b227e34d81a4501025e53beded P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:32.749270 10339 consensus_queue.cc:1035] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:32.753163 10339 consensus_queue.cc:1035] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:32.800005 10444 raft_consensus.cc:482] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:32.800160 10444 raft_consensus.cc:504] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:32.800566 10444 leader_election.cc:288] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:32.800968 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9de084b227e34d81a4501025e53beded" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:32.801009 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9de084b227e34d81a4501025e53beded" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:32.801097 8947 raft_consensus.cc:2454] T 9de084b227e34d81a4501025e53beded P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:32.801157 9080 raft_consensus.cc:2454] T 9de084b227e34d81a4501025e53beded P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:32.801416 6726 leader_election.cc:302] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:32.801589 10444 raft_consensus.cc:2781] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:32.801647 10444 raft_consensus.cc:482] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:32.801695 10444 raft_consensus.cc:3037] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.802639 10444 raft_consensus.cc:504] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:32.802973 10444 leader_election.cc:288] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:32.803329 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9de084b227e34d81a4501025e53beded" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:32.803391 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9de084b227e34d81a4501025e53beded" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:32.803442 8947 raft_consensus.cc:3037] T 9de084b227e34d81a4501025e53beded P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.803520 9080 raft_consensus.cc:3037] T 9de084b227e34d81a4501025e53beded P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.804260 8947 raft_consensus.cc:2454] T 9de084b227e34d81a4501025e53beded P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:32.804404 9080 raft_consensus.cc:2454] T 9de084b227e34d81a4501025e53beded P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:32.804572 6726 leader_election.cc:302] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:32.804749 10444 raft_consensus.cc:2781] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:32.804833 10444 raft_consensus.cc:686] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:32.804972 10444 consensus_queue.cc:237] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:32.807255 6383 catalog_manager.cc:5484] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:32.865233 10436 consensus_queue.cc:1035] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:32.870733 10148 consensus_queue.cc:1035] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:32.913226 6383 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:32.917420 8947 raft_consensus.cc:1261] T 9de084b227e34d81a4501025e53beded P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:32.917484 9080 raft_consensus.cc:1261] T 9de084b227e34d81a4501025e53beded P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:32.917927 10444 consensus_queue.cc:1035] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:32.918159 10345 consensus_queue.cc:1035] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:32.920876 6383 catalog_manager.cc:5576] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481) does not have the latest schema for tablet 9de084b227e34d81a4501025e53beded (table rebalancer_dynamic_table [id=bed9b6a9c9bf4c0fbd30c2988b7fc824]). Expected version 1 got 0
I20240503 13:59:32.925945 10461 tablet.cc:1722] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:32.926328 10461 tablet.cc:1620] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64: MemRowSet was empty: no flush needed.
I20240503 13:59:32.927521 6383 catalog_manager.cc:5576] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365) does not have the latest schema for tablet 9de084b227e34d81a4501025e53beded (table rebalancer_dynamic_table [id=bed9b6a9c9bf4c0fbd30c2988b7fc824]). Expected version 1 got 0
I20240503 13:59:32.928943 10196 consensus_queue.cc:1035] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:32.929492 6383 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet 9de084b227e34d81a4501025e53beded (table rebalancer_dynamic_table [id=bed9b6a9c9bf4c0fbd30c2988b7fc824]). Expected version 1 got 0
I20240503 13:59:32.931988 10419 tablet.cc:1722] T 9de084b227e34d81a4501025e53beded P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:32.932312 10419 tablet.cc:1620] T 9de084b227e34d81a4501025e53beded P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:32.934823 10465 tablet.cc:1722] T 9de084b227e34d81a4501025e53beded P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:32.935227 10465 tablet.cc:1620] T 9de084b227e34d81a4501025e53beded P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:32.936089 6386 catalog_manager.cc:5576] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481) does not have the latest schema for tablet 9de084b227e34d81a4501025e53beded (table rebalancer_dynamic_table [id=bed9b6a9c9bf4c0fbd30c2988b7fc824]). Expected version 1 got 0
I20240503 13:59:32.936583 6383 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=bed9b6a9c9bf4c0fbd30c2988b7fc824] alter complete (version 1)
I20240503 13:59:32.938305 10196 consensus_queue.cc:1035] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:32.989210 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:32.996276 6902 tablet_service.cc:1467] Processing CreateTablet for tablet 7a25af1e7ae64f4fad8ad82faa1c6eb6 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=22ad55d53adb4db5b7dd551f6384d11b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.996771 6902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a25af1e7ae64f4fad8ad82faa1c6eb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.998862 10467 tablet_bootstrap.cc:492] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:32.999477 9190 tablet_service.cc:1467] Processing CreateTablet for tablet f4cba4ca296e407c9520a6eb7419ed9d (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=22ad55d53adb4db5b7dd551f6384d11b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.999732 9191 tablet_service.cc:1467] Processing CreateTablet for tablet 174e5b7577d94d87b729d1735987a97a (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=22ad55d53adb4db5b7dd551f6384d11b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.999751 6508 tablet_service.cc:1467] Processing CreateTablet for tablet f4cba4ca296e407c9520a6eb7419ed9d (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=22ad55d53adb4db5b7dd551f6384d11b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.000072 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4cba4ca296e407c9520a6eb7419ed9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.000312 8927 tablet_service.cc:1467] Processing CreateTablet for tablet f4cba4ca296e407c9520a6eb7419ed9d (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=22ad55d53adb4db5b7dd551f6384d11b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.000372 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 7a25af1e7ae64f4fad8ad82faa1c6eb6 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=22ad55d53adb4db5b7dd551f6384d11b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.000579 8927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4cba4ca296e407c9520a6eb7419ed9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.000633 10467 tablet_bootstrap.cc:654] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.000839 8926 tablet_service.cc:1467] Processing CreateTablet for tablet 174e5b7577d94d87b729d1735987a97a (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=22ad55d53adb4db5b7dd551f6384d11b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.001098 8926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174e5b7577d94d87b729d1735987a97a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.002616 10468 tablet_bootstrap.cc:492] T f4cba4ca296e407c9520a6eb7419ed9d P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:33.002816 10397 tablet_bootstrap.cc:492] T f4cba4ca296e407c9520a6eb7419ed9d P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:33.003348 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a25af1e7ae64f4fad8ad82faa1c6eb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.003836 10397 tablet_bootstrap.cc:654] T f4cba4ca296e407c9520a6eb7419ed9d P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.003917 10468 tablet_bootstrap.cc:654] T f4cba4ca296e407c9520a6eb7419ed9d P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.005249 10397 tablet_bootstrap.cc:492] T f4cba4ca296e407c9520a6eb7419ed9d P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:32.999747 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4cba4ca296e407c9520a6eb7419ed9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.005344 10397 ts_tablet_manager.cc:1367] T f4cba4ca296e407c9520a6eb7419ed9d P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:33.005527 9191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174e5b7577d94d87b729d1735987a97a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.005859 10397 raft_consensus.cc:348] T f4cba4ca296e407c9520a6eb7419ed9d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:33.005981 10397 raft_consensus.cc:374] T f4cba4ca296e407c9520a6eb7419ed9d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.006016 10397 raft_consensus.cc:729] T f4cba4ca296e407c9520a6eb7419ed9d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.006306 9323 tablet_service.cc:1467] Processing CreateTablet for tablet 7a25af1e7ae64f4fad8ad82faa1c6eb6 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=22ad55d53adb4db5b7dd551f6384d11b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.006683 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a25af1e7ae64f4fad8ad82faa1c6eb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.007551 9322 tablet_service.cc:1467] Processing CreateTablet for tablet 174e5b7577d94d87b729d1735987a97a (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=22ad55d53adb4db5b7dd551f6384d11b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.007781 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174e5b7577d94d87b729d1735987a97a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.007920 10397 consensus_queue.cc:260] T f4cba4ca296e407c9520a6eb7419ed9d P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:33.008239 10397 ts_tablet_manager.cc:1398] T f4cba4ca296e407c9520a6eb7419ed9d P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:33.008388 10397 tablet_bootstrap.cc:492] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:33.009621 10397 tablet_bootstrap.cc:654] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.010973 10472 tablet_bootstrap.cc:492] T 174e5b7577d94d87b729d1735987a97a P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:33.011456 10397 tablet_bootstrap.cc:492] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:33.011533 10397 ts_tablet_manager.cc:1367] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:33.012041 10397 raft_consensus.cc:348] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:33.012144 10397 raft_consensus.cc:374] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.012182 10397 raft_consensus.cc:729] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.012300 10397 consensus_queue.cc:260] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:33.012594 10397 ts_tablet_manager.cc:1398] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:33.012763 10397 tablet_bootstrap.cc:492] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:33.013031 10468 tablet_bootstrap.cc:492] T f4cba4ca296e407c9520a6eb7419ed9d P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:33.013712 10397 tablet_bootstrap.cc:654] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.015925 10397 tablet_bootstrap.cc:492] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:33.016032 10397 ts_tablet_manager.cc:1367] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:33.016675 10468 ts_tablet_manager.cc:1367] T f4cba4ca296e407c9520a6eb7419ed9d P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20240503 13:59:33.017009 10397 raft_consensus.cc:348] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:33.017189 10397 raft_consensus.cc:374] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.017207 10468 raft_consensus.cc:348] T f4cba4ca296e407c9520a6eb7419ed9d P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:33.017274 10397 raft_consensus.cc:729] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.017328 10468 raft_consensus.cc:374] T f4cba4ca296e407c9520a6eb7419ed9d P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.017380 10468 raft_consensus.cc:729] T f4cba4ca296e407c9520a6eb7419ed9d P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.017513 10468 consensus_queue.cc:260] T f4cba4ca296e407c9520a6eb7419ed9d P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:33.017704 10397 consensus_queue.cc:260] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:33.018182 10468 ts_tablet_manager.cc:1398] T f4cba4ca296e407c9520a6eb7419ed9d P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:33.018507 10473 tablet_bootstrap.cc:492] T 174e5b7577d94d87b729d1735987a97a P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:33.018811 10397 ts_tablet_manager.cc:1398] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:33.019275 10472 tablet_bootstrap.cc:654] T 174e5b7577d94d87b729d1735987a97a P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.020679 10473 tablet_bootstrap.cc:654] T 174e5b7577d94d87b729d1735987a97a P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.022014 10472 tablet_bootstrap.cc:492] T 174e5b7577d94d87b729d1735987a97a P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:33.022145 10472 ts_tablet_manager.cc:1367] T 174e5b7577d94d87b729d1735987a97a P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 13:59:33.022208 10473 tablet_bootstrap.cc:492] T 174e5b7577d94d87b729d1735987a97a P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:33.022290 10473 ts_tablet_manager.cc:1367] T 174e5b7577d94d87b729d1735987a97a P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 13:59:33.022576 10472 raft_consensus.cc:348] T 174e5b7577d94d87b729d1735987a97a P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:33.022671 10472 raft_consensus.cc:374] T 174e5b7577d94d87b729d1735987a97a P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.022692 10472 raft_consensus.cc:729] T 174e5b7577d94d87b729d1735987a97a P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.022742 10473 raft_consensus.cc:348] T 174e5b7577d94d87b729d1735987a97a P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:33.022835 10473 raft_consensus.cc:374] T 174e5b7577d94d87b729d1735987a97a P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.022805 10472 consensus_queue.cc:260] T 174e5b7577d94d87b729d1735987a97a P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:33.022861 10473 raft_consensus.cc:729] T 174e5b7577d94d87b729d1735987a97a P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.022964 10473 consensus_queue.cc:260] T 174e5b7577d94d87b729d1735987a97a P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:33.023054 10472 ts_tablet_manager.cc:1398] T 174e5b7577d94d87b729d1735987a97a P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:33.023204 10472 tablet_bootstrap.cc:492] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:33.023268 10473 ts_tablet_manager.cc:1398] T 174e5b7577d94d87b729d1735987a97a P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:33.023512 10473 tablet_bootstrap.cc:492] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:33.024209 10472 tablet_bootstrap.cc:654] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.024681 10473 tablet_bootstrap.cc:654] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.032024 10467 tablet_bootstrap.cc:492] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:33.032124 10467 ts_tablet_manager.cc:1367] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.002s
I20240503 13:59:33.032212 10472 tablet_bootstrap.cc:492] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:33.032294 10472 ts_tablet_manager.cc:1367] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:59:33.032397 10473 tablet_bootstrap.cc:492] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:33.032471 10473 ts_tablet_manager.cc:1367] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240503 13:59:33.032578 10467 raft_consensus.cc:348] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:33.032682 10467 raft_consensus.cc:374] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.032712 10467 raft_consensus.cc:729] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.032738 10472 raft_consensus.cc:348] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:33.032830 10467 consensus_queue.cc:260] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:33.032891 10472 raft_consensus.cc:374] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.032928 10472 raft_consensus.cc:729] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.032876 10473 raft_consensus.cc:348] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:33.032987 10473 raft_consensus.cc:374] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.033017 10473 raft_consensus.cc:729] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.033085 10467 ts_tablet_manager.cc:1398] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:33.033051 10472 consensus_queue.cc:260] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:33.033124 10473 consensus_queue.cc:260] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:33.033367 10473 ts_tablet_manager.cc:1398] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:33.033394 10472 ts_tablet_manager.cc:1398] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:33.038851 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:33.044052 10461 tablet.cc:1722] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:33.044348 10461 tablet.cc:1620] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64: MemRowSet was empty: no flush needed.
I20240503 13:59:33.044768 10419 tablet.cc:1722] T 9de084b227e34d81a4501025e53beded P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:33.044770 10465 tablet.cc:1722] T 9de084b227e34d81a4501025e53beded P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:33.045061 10465 tablet.cc:1620] T 9de084b227e34d81a4501025e53beded P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:33.045061 10419 tablet.cc:1620] T 9de084b227e34d81a4501025e53beded P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:33.050335 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=bed9b6a9c9bf4c0fbd30c2988b7fc824] alter complete (version 2)
I20240503 13:59:33.104991 10339 raft_consensus.cc:482] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:33.105122 10339 raft_consensus.cc:504] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:33.105535 10339 leader_election.cc:288] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:33.106024 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a25af1e7ae64f4fad8ad82faa1c6eb6" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:33.106061 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a25af1e7ae64f4fad8ad82faa1c6eb6" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:33.106165 6920 raft_consensus.cc:2454] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:33.106189 8947 raft_consensus.cc:2454] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:33.106508 9278 leader_election.cc:302] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:33.106750 10339 raft_consensus.cc:2781] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:33.106807 10339 raft_consensus.cc:482] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:33.106846 10339 raft_consensus.cc:3037] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.107930 10339 raft_consensus.cc:504] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:33.108274 10339 leader_election.cc:288] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:33.108588 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a25af1e7ae64f4fad8ad82faa1c6eb6" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:33.108633 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a25af1e7ae64f4fad8ad82faa1c6eb6" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:33.108767 6920 raft_consensus.cc:3037] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.108794 8948 raft_consensus.cc:3037] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.109591 8948 raft_consensus.cc:2454] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:33.109644 6920 raft_consensus.cc:2454] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:33.109864 9278 leader_election.cc:302] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 0d77752181e944698e809e325e5937e6; no voters:
I20240503 13:59:33.110015 10339 raft_consensus.cc:2781] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:33.110118 10339 raft_consensus.cc:686] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:33.110256 10339 consensus_queue.cc:237] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:33.111992 10456 raft_consensus.cc:482] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:33.112087 10456 raft_consensus.cc:504] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:33.112282 6380 catalog_manager.cc:5484] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:33.112465 10456 leader_election.cc:288] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:33.112766 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174e5b7577d94d87b729d1735987a97a" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:33.112876 9343 raft_consensus.cc:2454] T 174e5b7577d94d87b729d1735987a97a P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:33.113034 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174e5b7577d94d87b729d1735987a97a" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:33.113186 9205 raft_consensus.cc:2454] T 174e5b7577d94d87b729d1735987a97a P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:33.113490 8883 leader_election.cc:302] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:33.113656 10456 raft_consensus.cc:2781] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:33.113721 10456 raft_consensus.cc:482] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:33.113767 10456 raft_consensus.cc:3037] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.114760 10456 raft_consensus.cc:504] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:33.115095 10456 leader_election.cc:288] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:33.115834 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174e5b7577d94d87b729d1735987a97a" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:33.115947 9205 raft_consensus.cc:3037] T 174e5b7577d94d87b729d1735987a97a P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.116827 9205 raft_consensus.cc:2454] T 174e5b7577d94d87b729d1735987a97a P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:33.117271 8883 leader_election.cc:302] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:33.117426 10456 raft_consensus.cc:2781] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:33.117414 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174e5b7577d94d87b729d1735987a97a" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:33.117493 10456 raft_consensus.cc:686] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 13:59:33.117506 9343 raft_consensus.cc:3037] T 174e5b7577d94d87b729d1735987a97a P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.117597 10456 consensus_queue.cc:237] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:33.118297 9343 raft_consensus.cc:2454] T 174e5b7577d94d87b729d1735987a97a P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:33.119859 6380 catalog_manager.cc:5484] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:33.151604 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:33.156431 10461 tablet.cc:1722] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:33.157052 10465 tablet.cc:1722] T 9de084b227e34d81a4501025e53beded P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:33.158972 10419 tablet.cc:1722] T 9de084b227e34d81a4501025e53beded P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:33.162451 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=bed9b6a9c9bf4c0fbd30c2988b7fc824] alter complete (version 3)
I20240503 13:59:33.263998 6380 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:33.264181 6380 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:33.266227 6380 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 9de084b227e34d81a4501025e53beded
I20240503 13:59:33.266784 6769 tablet_service.cc:1514] Processing DeleteTablet for tablet 9de084b227e34d81a4501025e53beded with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:33 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:33.266834 8928 tablet_service.cc:1514] Processing DeleteTablet for tablet 9de084b227e34d81a4501025e53beded with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:33 UTC) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:33.267164 10483 tablet_replica.cc:330] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:33.267305 10484 tablet_replica.cc:330] T 9de084b227e34d81a4501025e53beded P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:33.267369 10483 raft_consensus.cc:2229] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:33.267354 9058 tablet_service.cc:1514] Processing DeleteTablet for tablet 9de084b227e34d81a4501025e53beded with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:33 UTC) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:33.267486 10484 raft_consensus.cc:2229] T 9de084b227e34d81a4501025e53beded P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:33.267589 10483 raft_consensus.cc:2258] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:33.267616 10484 raft_consensus.cc:2258] T 9de084b227e34d81a4501025e53beded P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:33.268133 10483 ts_tablet_manager.cc:1875] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:33.268246 10484 ts_tablet_manager.cc:1875] T 9de084b227e34d81a4501025e53beded P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:33.269973 10485 tablet_replica.cc:330] T 9de084b227e34d81a4501025e53beded P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:33.270129 10485 raft_consensus.cc:2229] T 9de084b227e34d81a4501025e53beded P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:33.270262 10485 raft_consensus.cc:2258] T 9de084b227e34d81a4501025e53beded P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:33.270354 10483 ts_tablet_manager.cc:1888] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:33.270654 10483 log.cc:1196] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/9de084b227e34d81a4501025e53beded
I20240503 13:59:33.270967 10484 ts_tablet_manager.cc:1888] T 9de084b227e34d81a4501025e53beded P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:33.271041 10483 ts_tablet_manager.cc:1909] T 9de084b227e34d81a4501025e53beded P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:33.271070 10484 log.cc:1196] T 9de084b227e34d81a4501025e53beded P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/9de084b227e34d81a4501025e53beded
I20240503 13:59:33.271080 10485 ts_tablet_manager.cc:1875] T 9de084b227e34d81a4501025e53beded P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:33.271378 10484 ts_tablet_manager.cc:1909] T 9de084b227e34d81a4501025e53beded P 06ca8dbc004c42b99faa527a4e6f9545: Deleting consensus metadata
I20240503 13:59:33.271807 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet 9de084b227e34d81a4501025e53beded (table rebalancer_dynamic_table_new_name [id=bed9b6a9c9bf4c0fbd30c2988b7fc824]) successfully deleted
I20240503 13:59:33.272001 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 9de084b227e34d81a4501025e53beded (table rebalancer_dynamic_table_new_name [id=bed9b6a9c9bf4c0fbd30c2988b7fc824]) successfully deleted
I20240503 13:59:33.273330 10485 ts_tablet_manager.cc:1888] T 9de084b227e34d81a4501025e53beded P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:33.273409 10485 log.cc:1196] T 9de084b227e34d81a4501025e53beded P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/9de084b227e34d81a4501025e53beded
I20240503 13:59:33.273708 10485 ts_tablet_manager.cc:1909] T 9de084b227e34d81a4501025e53beded P 8d574fb4a89d4eea947c859470d44dd4: Deleting consensus metadata
I20240503 13:59:33.274489 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet 9de084b227e34d81a4501025e53beded (table rebalancer_dynamic_table_new_name [id=bed9b6a9c9bf4c0fbd30c2988b7fc824]) successfully deleted
I20240503 13:59:33.368327 6380 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:33.374502 9323 tablet_service.cc:1467] Processing CreateTablet for tablet 4d023ca01b6a405ab2d115df8e3e1271 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1cbda783fff744fba4da8b42484e364c]), partition=
I20240503 13:59:33.374502 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 4d023ca01b6a405ab2d115df8e3e1271 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1cbda783fff744fba4da8b42484e364c]), partition=
I20240503 13:59:33.374794 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d023ca01b6a405ab2d115df8e3e1271. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.374805 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d023ca01b6a405ab2d115df8e3e1271. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.377143 10473 tablet_bootstrap.cc:492] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:33.377700 10487 tablet_bootstrap.cc:492] T 4d023ca01b6a405ab2d115df8e3e1271 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:33.377996 10473 tablet_bootstrap.cc:654] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.378691 10487 tablet_bootstrap.cc:654] T 4d023ca01b6a405ab2d115df8e3e1271 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.378819 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 4d023ca01b6a405ab2d115df8e3e1271 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1cbda783fff744fba4da8b42484e364c]), partition=
I20240503 13:59:33.379055 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d023ca01b6a405ab2d115df8e3e1271. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.379823 10487 tablet_bootstrap.cc:492] T 4d023ca01b6a405ab2d115df8e3e1271 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:33.379889 10487 ts_tablet_manager.cc:1367] T 4d023ca01b6a405ab2d115df8e3e1271 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:33.380232 10487 raft_consensus.cc:348] T 4d023ca01b6a405ab2d115df8e3e1271 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:33.380311 10487 raft_consensus.cc:374] T 4d023ca01b6a405ab2d115df8e3e1271 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.380333 10487 raft_consensus.cc:729] T 4d023ca01b6a405ab2d115df8e3e1271 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.380412 10487 consensus_queue.cc:260] T 4d023ca01b6a405ab2d115df8e3e1271 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:33.380764 10487 ts_tablet_manager.cc:1398] T 4d023ca01b6a405ab2d115df8e3e1271 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:33.381373 10468 tablet_bootstrap.cc:492] T 4d023ca01b6a405ab2d115df8e3e1271 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:33.382294 10468 tablet_bootstrap.cc:654] T 4d023ca01b6a405ab2d115df8e3e1271 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.382438 10473 tablet_bootstrap.cc:492] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:33.382529 10473 ts_tablet_manager.cc:1367] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 13:59:33.383360 10473 raft_consensus.cc:348] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:33.383486 10473 raft_consensus.cc:374] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.383525 10473 raft_consensus.cc:729] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.383644 10473 consensus_queue.cc:260] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:33.383955 10473 ts_tablet_manager.cc:1398] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:33.386407 10468 tablet_bootstrap.cc:492] T 4d023ca01b6a405ab2d115df8e3e1271 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:33.386507 10468 ts_tablet_manager.cc:1367] T 4d023ca01b6a405ab2d115df8e3e1271 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 13:59:33.386996 10468 raft_consensus.cc:348] T 4d023ca01b6a405ab2d115df8e3e1271 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:33.387173 10468 raft_consensus.cc:374] T 4d023ca01b6a405ab2d115df8e3e1271 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.387235 10468 raft_consensus.cc:729] T 4d023ca01b6a405ab2d115df8e3e1271 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.387396 10468 consensus_queue.cc:260] T 4d023ca01b6a405ab2d115df8e3e1271 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:33.387786 10468 ts_tablet_manager.cc:1398] T 4d023ca01b6a405ab2d115df8e3e1271 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:33.396376 10443 raft_consensus.cc:482] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:33.396520 10443 raft_consensus.cc:504] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:33.397228 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4cba4ca296e407c9520a6eb7419ed9d" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:33.397380 8947 raft_consensus.cc:2454] T f4cba4ca296e407c9520a6eb7419ed9d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:33.397485 10443 leader_election.cc:288] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:33.397552 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4cba4ca296e407c9520a6eb7419ed9d" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:33.397629 9147 leader_election.cc:302] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:33.397679 6528 raft_consensus.cc:2454] T f4cba4ca296e407c9520a6eb7419ed9d P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:33.397768 10443 raft_consensus.cc:2781] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:33.397814 10443 raft_consensus.cc:482] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:33.397846 10443 raft_consensus.cc:3037] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.398845 10443 raft_consensus.cc:504] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:33.399230 10443 leader_election.cc:288] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Requested vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:33.399590 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4cba4ca296e407c9520a6eb7419ed9d" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:33.399621 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4cba4ca296e407c9520a6eb7419ed9d" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:33.399717 8947 raft_consensus.cc:3037] T f4cba4ca296e407c9520a6eb7419ed9d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.399717 6528 raft_consensus.cc:3037] T f4cba4ca296e407c9520a6eb7419ed9d P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.400681 6528 raft_consensus.cc:2454] T f4cba4ca296e407c9520a6eb7419ed9d P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:33.400748 8947 raft_consensus.cc:2454] T f4cba4ca296e407c9520a6eb7419ed9d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:33.400974 9147 leader_election.cc:302] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:33.401132 10443 raft_consensus.cc:2781] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:33.401300 10443 raft_consensus.cc:686] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 13:59:33.401434 10443 consensus_queue.cc:237] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:33.403241 6380 catalog_manager.cc:5484] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:33.426676 10486 raft_consensus.cc:482] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:33.426841 10486 raft_consensus.cc:504] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:33.427309 10486 leader_election.cc:288] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:33.427803 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d023ca01b6a405ab2d115df8e3e1271" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:33.427960 6658 raft_consensus.cc:2454] T 4d023ca01b6a405ab2d115df8e3e1271 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:33.428284 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d023ca01b6a405ab2d115df8e3e1271" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:33.428333 9279 leader_election.cc:302] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 2145d15ab2d74c0287839266f1411a43; no voters:
I20240503 13:59:33.428470 6528 raft_consensus.cc:2454] T 4d023ca01b6a405ab2d115df8e3e1271 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:33.428550 10486 raft_consensus.cc:2781] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:33.428612 10486 raft_consensus.cc:482] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:33.428659 10486 raft_consensus.cc:3037] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.429981 10486 raft_consensus.cc:504] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:33.430404 10486 leader_election.cc:288] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:33.430650 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d023ca01b6a405ab2d115df8e3e1271" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:33.430763 6528 raft_consensus.cc:3037] T 4d023ca01b6a405ab2d115df8e3e1271 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.430816 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d023ca01b6a405ab2d115df8e3e1271" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:33.430924 6658 raft_consensus.cc:3037] T 4d023ca01b6a405ab2d115df8e3e1271 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.431623 6528 raft_consensus.cc:2454] T 4d023ca01b6a405ab2d115df8e3e1271 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:33.431710 6658 raft_consensus.cc:2454] T 4d023ca01b6a405ab2d115df8e3e1271 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:33.431872 9278 leader_election.cc:302] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:33.432240 10486 raft_consensus.cc:2781] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:33.432339 10486 raft_consensus.cc:686] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:33.432482 10486 consensus_queue.cc:237] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:33.434959 6380 catalog_manager.cc:5484] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:33.515239 10148 consensus_queue.cc:1035] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:33.519240 10148 consensus_queue.cc:1035] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:33.524188 10339 consensus_queue.cc:1035] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:33.527637 10339 consensus_queue.cc:1035] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:33.540787 6383 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:33.544641 6528 raft_consensus.cc:1261] T 4d023ca01b6a405ab2d115df8e3e1271 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:33.544639 6658 raft_consensus.cc:1261] T 4d023ca01b6a405ab2d115df8e3e1271 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:33.545074 10339 consensus_queue.cc:1035] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:33.545214 10249 consensus_queue.cc:1035] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:33.548238 6383 catalog_manager.cc:5576] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119) does not have the latest schema for tablet 4d023ca01b6a405ab2d115df8e3e1271 (table rebalancer_dynamic_table [id=1cbda783fff744fba4da8b42484e364c]). Expected version 1 got 0
I20240503 13:59:33.548835 6380 catalog_manager.cc:5576] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021) does not have the latest schema for tablet 4d023ca01b6a405ab2d115df8e3e1271 (table rebalancer_dynamic_table [id=1cbda783fff744fba4da8b42484e364c]). Expected version 1 got 0
I20240503 13:59:33.549270 6386 catalog_manager.cc:5576] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931) does not have the latest schema for tablet 4d023ca01b6a405ab2d115df8e3e1271 (table rebalancer_dynamic_table [id=1cbda783fff744fba4da8b42484e364c]). Expected version 1 got 0
I20240503 13:59:33.549993 10505 tablet.cc:1722] T 4d023ca01b6a405ab2d115df8e3e1271 P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:33.550753 10505 tablet.cc:1620] T 4d023ca01b6a405ab2d115df8e3e1271 P 5f46e19644fd4f4b825c86ba6af768da: MemRowSet was empty: no flush needed.
I20240503 13:59:33.551134 10504 tablet.cc:1722] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:33.551486 10504 tablet.cc:1620] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:33.552042 6386 catalog_manager.cc:5576] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021) does not have the latest schema for tablet 4d023ca01b6a405ab2d115df8e3e1271 (table rebalancer_dynamic_table [id=1cbda783fff744fba4da8b42484e364c]). Expected version 1 got 0
I20240503 13:59:33.553943 6380 catalog_manager.cc:5576] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119) does not have the latest schema for tablet 4d023ca01b6a405ab2d115df8e3e1271 (table rebalancer_dynamic_table [id=1cbda783fff744fba4da8b42484e364c]). Expected version 1 got 0
I20240503 13:59:33.555069 10506 tablet.cc:1722] T 4d023ca01b6a405ab2d115df8e3e1271 P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:33.555577 10506 tablet.cc:1620] T 4d023ca01b6a405ab2d115df8e3e1271 P 2145d15ab2d74c0287839266f1411a43: MemRowSet was empty: no flush needed.
I20240503 13:59:33.558140 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=1cbda783fff744fba4da8b42484e364c] alter complete (version 1)
I20240503 13:59:33.662832 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:33.667837 10504 tablet.cc:1722] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:33.668180 10504 tablet.cc:1620] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:33.668323 10506 tablet.cc:1722] T 4d023ca01b6a405ab2d115df8e3e1271 P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:33.668453 10505 tablet.cc:1722] T 4d023ca01b6a405ab2d115df8e3e1271 P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:33.668759 10505 tablet.cc:1620] T 4d023ca01b6a405ab2d115df8e3e1271 P 5f46e19644fd4f4b825c86ba6af768da: MemRowSet was empty: no flush needed.
I20240503 13:59:33.668771 10506 tablet.cc:1620] T 4d023ca01b6a405ab2d115df8e3e1271 P 2145d15ab2d74c0287839266f1411a43: MemRowSet was empty: no flush needed.
I20240503 13:59:33.673411 6383 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=1cbda783fff744fba4da8b42484e364c] alter complete (version 2)
I20240503 13:59:33.775575 6383 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:33.781349 10504 tablet.cc:1722] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:33.782024 10505 tablet.cc:1722] T 4d023ca01b6a405ab2d115df8e3e1271 P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:33.782642 10506 tablet.cc:1722] T 4d023ca01b6a405ab2d115df8e3e1271 P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:33.787514 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=1cbda783fff744fba4da8b42484e364c] alter complete (version 3)
I20240503 13:59:33.829977 10443 consensus_queue.cc:1035] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:33.833652 10443 consensus_queue.cc:1035] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:33.890352 6380 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:33.890525 6380 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:33.892714 6380 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 4d023ca01b6a405ab2d115df8e3e1271
I20240503 13:59:33.893296 9323 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d023ca01b6a405ab2d115df8e3e1271 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:33 UTC) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:33.893393 6640 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d023ca01b6a405ab2d115df8e3e1271 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:33 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:33.893687 10511 tablet_replica.cc:330] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:33.893800 10512 tablet_replica.cc:330] T 4d023ca01b6a405ab2d115df8e3e1271 P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:33.893793 6508 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d023ca01b6a405ab2d115df8e3e1271 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:33 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:33.893850 10511 raft_consensus.cc:2229] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:33.894070 10512 raft_consensus.cc:2229] T 4d023ca01b6a405ab2d115df8e3e1271 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:33.894165 10513 tablet_replica.cc:330] T 4d023ca01b6a405ab2d115df8e3e1271 P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:33.894222 10511 raft_consensus.cc:2258] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:33.894284 10512 raft_consensus.cc:2258] T 4d023ca01b6a405ab2d115df8e3e1271 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:33.894404 10513 raft_consensus.cc:2229] T 4d023ca01b6a405ab2d115df8e3e1271 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:33.894554 10513 raft_consensus.cc:2258] T 4d023ca01b6a405ab2d115df8e3e1271 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:33.894882 10511 ts_tablet_manager.cc:1875] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:33.895251 10513 ts_tablet_manager.cc:1875] T 4d023ca01b6a405ab2d115df8e3e1271 P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:33.895483 10512 ts_tablet_manager.cc:1875] T 4d023ca01b6a405ab2d115df8e3e1271 P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:33.897353 10511 ts_tablet_manager.cc:1888] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:33.897498 10511 log.cc:1196] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/4d023ca01b6a405ab2d115df8e3e1271
I20240503 13:59:33.897703 10513 ts_tablet_manager.cc:1888] T 4d023ca01b6a405ab2d115df8e3e1271 P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:33.897783 10513 log.cc:1196] T 4d023ca01b6a405ab2d115df8e3e1271 P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/4d023ca01b6a405ab2d115df8e3e1271
I20240503 13:59:33.897806 10511 ts_tablet_manager.cc:1909] T 4d023ca01b6a405ab2d115df8e3e1271 P 0d77752181e944698e809e325e5937e6: Deleting consensus metadata
I20240503 13:59:33.897866 10512 ts_tablet_manager.cc:1888] T 4d023ca01b6a405ab2d115df8e3e1271 P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:33.897969 10512 log.cc:1196] T 4d023ca01b6a405ab2d115df8e3e1271 P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/4d023ca01b6a405ab2d115df8e3e1271
I20240503 13:59:33.898218 10513 ts_tablet_manager.cc:1909] T 4d023ca01b6a405ab2d115df8e3e1271 P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:33.898450 10512 ts_tablet_manager.cc:1909] T 4d023ca01b6a405ab2d115df8e3e1271 P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:33.898773 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet 4d023ca01b6a405ab2d115df8e3e1271 (table rebalancer_dynamic_table_new_name [id=1cbda783fff744fba4da8b42484e364c]) successfully deleted
I20240503 13:59:33.898957 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 4d023ca01b6a405ab2d115df8e3e1271 (table rebalancer_dynamic_table_new_name [id=1cbda783fff744fba4da8b42484e364c]) successfully deleted
I20240503 13:59:33.899489 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet 4d023ca01b6a405ab2d115df8e3e1271 (table rebalancer_dynamic_table_new_name [id=1cbda783fff744fba4da8b42484e364c]) successfully deleted
I20240503 13:59:33.963110 6380 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:33.969460 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 8482be0fb2784324b77b272e40f57d6b (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=25d54e91bed8402eb84d2c36a0601777]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.969719 9323 tablet_service.cc:1467] Processing CreateTablet for tablet 8482be0fb2784324b77b272e40f57d6b (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=25d54e91bed8402eb84d2c36a0601777]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.969812 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8482be0fb2784324b77b272e40f57d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.970013 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8482be0fb2784324b77b272e40f57d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.970355 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 8482be0fb2784324b77b272e40f57d6b (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=25d54e91bed8402eb84d2c36a0601777]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.970597 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8482be0fb2784324b77b272e40f57d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.972723 9322 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0daf41253f4678b158e3ccbb7b0c98 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=25d54e91bed8402eb84d2c36a0601777]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.972949 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f0daf41253f4678b158e3ccbb7b0c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.973034 9191 tablet_service.cc:1467] Processing CreateTablet for tablet ee295fdb8a2e4990a1e0b47dc30cd56b (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=25d54e91bed8402eb84d2c36a0601777]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.973186 10515 tablet_bootstrap.cc:492] T 8482be0fb2784324b77b272e40f57d6b P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:33.973330 9191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee295fdb8a2e4990a1e0b47dc30cd56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.973034 9190 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0daf41253f4678b158e3ccbb7b0c98 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=25d54e91bed8402eb84d2c36a0601777]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.973881 10517 tablet_bootstrap.cc:492] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:33.974165 10515 tablet_bootstrap.cc:654] T 8482be0fb2784324b77b272e40f57d6b P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.974553 8926 tablet_service.cc:1467] Processing CreateTablet for tablet ee295fdb8a2e4990a1e0b47dc30cd56b (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=25d54e91bed8402eb84d2c36a0601777]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.974764 8926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee295fdb8a2e4990a1e0b47dc30cd56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.974934 10517 tablet_bootstrap.cc:654] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.977578 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f0daf41253f4678b158e3ccbb7b0c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.978821 10520 tablet_bootstrap.cc:492] T ee295fdb8a2e4990a1e0b47dc30cd56b P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:33.979559 10517 tablet_bootstrap.cc:492] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:33.979666 10517 ts_tablet_manager.cc:1367] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240503 13:59:33.979878 10520 tablet_bootstrap.cc:654] T ee295fdb8a2e4990a1e0b47dc30cd56b P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.980213 10517 raft_consensus.cc:348] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:33.980445 10517 raft_consensus.cc:374] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.980501 10517 raft_consensus.cc:729] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.980667 10517 consensus_queue.cc:260] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:33.980726 6902 tablet_service.cc:1467] Processing CreateTablet for tablet ee295fdb8a2e4990a1e0b47dc30cd56b (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=25d54e91bed8402eb84d2c36a0601777]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.981009 6902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee295fdb8a2e4990a1e0b47dc30cd56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.981029 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0daf41253f4678b158e3ccbb7b0c98 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=25d54e91bed8402eb84d2c36a0601777]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.981158 10517 ts_tablet_manager.cc:1398] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 13:59:33.981232 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f0daf41253f4678b158e3ccbb7b0c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.983395 10522 tablet_bootstrap.cc:492] T ee295fdb8a2e4990a1e0b47dc30cd56b P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:33.983664 10516 tablet_bootstrap.cc:492] T 8482be0fb2784324b77b272e40f57d6b P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:33.985033 10522 tablet_bootstrap.cc:654] T ee295fdb8a2e4990a1e0b47dc30cd56b P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.985476 10516 tablet_bootstrap.cc:654] T 8482be0fb2784324b77b272e40f57d6b P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.986338 10522 tablet_bootstrap.cc:492] T ee295fdb8a2e4990a1e0b47dc30cd56b P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:33.986433 10522 ts_tablet_manager.cc:1367] T ee295fdb8a2e4990a1e0b47dc30cd56b P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:33.986608 10515 tablet_bootstrap.cc:492] T 8482be0fb2784324b77b272e40f57d6b P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:33.986688 10515 ts_tablet_manager.cc:1367] T 8482be0fb2784324b77b272e40f57d6b P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 13:59:33.986747 10516 tablet_bootstrap.cc:492] T 8482be0fb2784324b77b272e40f57d6b P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:33.986824 10516 ts_tablet_manager.cc:1367] T 8482be0fb2784324b77b272e40f57d6b P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:33.986958 10522 raft_consensus.cc:348] T ee295fdb8a2e4990a1e0b47dc30cd56b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:33.987079 10522 raft_consensus.cc:374] T ee295fdb8a2e4990a1e0b47dc30cd56b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.987133 10522 raft_consensus.cc:729] T ee295fdb8a2e4990a1e0b47dc30cd56b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.987176 10515 raft_consensus.cc:348] T 8482be0fb2784324b77b272e40f57d6b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:33.987259 10522 consensus_queue.cc:260] T ee295fdb8a2e4990a1e0b47dc30cd56b P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:33.987244 10516 raft_consensus.cc:348] T 8482be0fb2784324b77b272e40f57d6b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:33.987373 10516 raft_consensus.cc:374] T 8482be0fb2784324b77b272e40f57d6b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.987407 10516 raft_consensus.cc:729] T 8482be0fb2784324b77b272e40f57d6b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.987577 10522 ts_tablet_manager.cc:1398] T ee295fdb8a2e4990a1e0b47dc30cd56b P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:33.987540 10516 consensus_queue.cc:260] T 8482be0fb2784324b77b272e40f57d6b P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:33.988058 10520 tablet_bootstrap.cc:492] T ee295fdb8a2e4990a1e0b47dc30cd56b P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:33.988142 10520 ts_tablet_manager.cc:1367] T ee295fdb8a2e4990a1e0b47dc30cd56b P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240503 13:59:33.988646 10515 raft_consensus.cc:374] T 8482be0fb2784324b77b272e40f57d6b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.988617 10520 raft_consensus.cc:348] T ee295fdb8a2e4990a1e0b47dc30cd56b P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:33.988703 10515 raft_consensus.cc:729] T 8482be0fb2784324b77b272e40f57d6b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.988723 10520 raft_consensus.cc:374] T ee295fdb8a2e4990a1e0b47dc30cd56b P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.988756 10520 raft_consensus.cc:729] T ee295fdb8a2e4990a1e0b47dc30cd56b P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.988830 10515 consensus_queue.cc:260] T 8482be0fb2784324b77b272e40f57d6b P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:33.988867 10520 consensus_queue.cc:260] T ee295fdb8a2e4990a1e0b47dc30cd56b P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:33.989135 10520 ts_tablet_manager.cc:1398] T ee295fdb8a2e4990a1e0b47dc30cd56b P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:33.989145 10515 ts_tablet_manager.cc:1398] T 8482be0fb2784324b77b272e40f57d6b P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:33.989310 10515 tablet_bootstrap.cc:492] T ee295fdb8a2e4990a1e0b47dc30cd56b P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:33.989327 10520 tablet_bootstrap.cc:492] T 9f0daf41253f4678b158e3ccbb7b0c98 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:33.990298 10515 tablet_bootstrap.cc:654] T ee295fdb8a2e4990a1e0b47dc30cd56b P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.990336 10520 tablet_bootstrap.cc:654] T 9f0daf41253f4678b158e3ccbb7b0c98 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.992122 10515 tablet_bootstrap.cc:492] T ee295fdb8a2e4990a1e0b47dc30cd56b P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:33.992225 10515 ts_tablet_manager.cc:1367] T ee295fdb8a2e4990a1e0b47dc30cd56b P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:59:33.992389 10520 tablet_bootstrap.cc:492] T 9f0daf41253f4678b158e3ccbb7b0c98 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:33.992456 10520 ts_tablet_manager.cc:1367] T 9f0daf41253f4678b158e3ccbb7b0c98 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:59:33.992662 10515 raft_consensus.cc:348] T ee295fdb8a2e4990a1e0b47dc30cd56b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:33.992751 10515 raft_consensus.cc:374] T ee295fdb8a2e4990a1e0b47dc30cd56b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.992782 10515 raft_consensus.cc:729] T ee295fdb8a2e4990a1e0b47dc30cd56b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.992868 10520 raft_consensus.cc:348] T 9f0daf41253f4678b158e3ccbb7b0c98 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:33.992955 10520 raft_consensus.cc:374] T 9f0daf41253f4678b158e3ccbb7b0c98 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.992916 10515 consensus_queue.cc:260] T ee295fdb8a2e4990a1e0b47dc30cd56b P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:33.992985 10520 raft_consensus.cc:729] T 9f0daf41253f4678b158e3ccbb7b0c98 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.993171 10520 consensus_queue.cc:260] T 9f0daf41253f4678b158e3ccbb7b0c98 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:33.993290 10515 ts_tablet_manager.cc:1398] T ee295fdb8a2e4990a1e0b47dc30cd56b P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:33.993467 10515 tablet_bootstrap.cc:492] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:33.993484 10520 ts_tablet_manager.cc:1398] T 9f0daf41253f4678b158e3ccbb7b0c98 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:33.993770 10516 ts_tablet_manager.cc:1398] T 8482be0fb2784324b77b272e40f57d6b P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 13:59:33.993979 10516 tablet_bootstrap.cc:492] T 9f0daf41253f4678b158e3ccbb7b0c98 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:33.994357 10515 tablet_bootstrap.cc:654] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.994415 6386 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:33.994961 10516 tablet_bootstrap.cc:654] T 9f0daf41253f4678b158e3ccbb7b0c98 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.997828 10516 tablet_bootstrap.cc:492] T 9f0daf41253f4678b158e3ccbb7b0c98 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:33.997920 10516 ts_tablet_manager.cc:1367] T 9f0daf41253f4678b158e3ccbb7b0c98 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 13:59:33.998378 10516 raft_consensus.cc:348] T 9f0daf41253f4678b158e3ccbb7b0c98 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:33.998474 10516 raft_consensus.cc:374] T 9f0daf41253f4678b158e3ccbb7b0c98 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.998508 10516 raft_consensus.cc:729] T 9f0daf41253f4678b158e3ccbb7b0c98 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.998643 10516 consensus_queue.cc:260] T 9f0daf41253f4678b158e3ccbb7b0c98 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:33.998893 10516 ts_tablet_manager.cc:1398] T 9f0daf41253f4678b158e3ccbb7b0c98 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 13:59:34.003759 8928 tablet_service.cc:1467] Processing CreateTablet for tablet d5ad9e6c93b046338e5b0003821e9803 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e85ab5044d6344f9ad4df66b7cdddcef]), partition=
I20240503 13:59:34.004002 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5ad9e6c93b046338e5b0003821e9803. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.004523 9058 tablet_service.cc:1467] Processing CreateTablet for tablet d5ad9e6c93b046338e5b0003821e9803 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e85ab5044d6344f9ad4df66b7cdddcef]), partition=
I20240503 13:59:34.004771 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5ad9e6c93b046338e5b0003821e9803. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.006608 9190 tablet_service.cc:1467] Processing CreateTablet for tablet d5ad9e6c93b046338e5b0003821e9803 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e85ab5044d6344f9ad4df66b7cdddcef]), partition=
I20240503 13:59:34.006814 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5ad9e6c93b046338e5b0003821e9803. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.007126 10517 tablet_bootstrap.cc:492] T d5ad9e6c93b046338e5b0003821e9803 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:34.008064 10517 tablet_bootstrap.cc:654] T d5ad9e6c93b046338e5b0003821e9803 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.008957 10520 tablet_bootstrap.cc:492] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:34.009884 10520 tablet_bootstrap.cc:654] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.011243 10520 tablet_bootstrap.cc:492] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:34.011358 10520 ts_tablet_manager.cc:1367] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:34.011785 10520 raft_consensus.cc:348] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:34.011886 10520 raft_consensus.cc:374] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.011914 10520 raft_consensus.cc:729] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.012033 10520 consensus_queue.cc:260] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:34.012320 10520 ts_tablet_manager.cc:1398] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:34.013303 10517 tablet_bootstrap.cc:492] T d5ad9e6c93b046338e5b0003821e9803 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:34.013396 10517 ts_tablet_manager.cc:1367] T d5ad9e6c93b046338e5b0003821e9803 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:59:34.013854 10517 raft_consensus.cc:348] T d5ad9e6c93b046338e5b0003821e9803 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:34.013947 10517 raft_consensus.cc:374] T d5ad9e6c93b046338e5b0003821e9803 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.013981 10517 raft_consensus.cc:729] T d5ad9e6c93b046338e5b0003821e9803 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.014117 10517 consensus_queue.cc:260] T d5ad9e6c93b046338e5b0003821e9803 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:34.014437 10517 ts_tablet_manager.cc:1398] T d5ad9e6c93b046338e5b0003821e9803 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:34.014901 10515 tablet_bootstrap.cc:492] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:34.014986 10515 ts_tablet_manager.cc:1367] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.001s
I20240503 13:59:34.015502 10515 raft_consensus.cc:348] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:34.015614 10515 raft_consensus.cc:374] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.015643 10515 raft_consensus.cc:729] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.015762 10515 consensus_queue.cc:260] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:34.016016 10515 ts_tablet_manager.cc:1398] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:34.016170 10515 tablet_bootstrap.cc:492] T d5ad9e6c93b046338e5b0003821e9803 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:34.017160 10515 tablet_bootstrap.cc:654] T d5ad9e6c93b046338e5b0003821e9803 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.019044 10515 tablet_bootstrap.cc:492] T d5ad9e6c93b046338e5b0003821e9803 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:34.019196 10515 ts_tablet_manager.cc:1367] T d5ad9e6c93b046338e5b0003821e9803 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:34.019647 10515 raft_consensus.cc:348] T d5ad9e6c93b046338e5b0003821e9803 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:34.019748 10515 raft_consensus.cc:374] T d5ad9e6c93b046338e5b0003821e9803 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.019779 10515 raft_consensus.cc:729] T d5ad9e6c93b046338e5b0003821e9803 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.019922 10515 consensus_queue.cc:260] T d5ad9e6c93b046338e5b0003821e9803 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:34.020278 10515 ts_tablet_manager.cc:1398] T d5ad9e6c93b046338e5b0003821e9803 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:34.031188 10527 raft_consensus.cc:482] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:34.031323 10527 raft_consensus.cc:504] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:34.031709 10527 leader_election.cc:288] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:34.032058 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8482be0fb2784324b77b272e40f57d6b" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:34.032192 8947 raft_consensus.cc:2454] T 8482be0fb2784324b77b272e40f57d6b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:34.032081 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8482be0fb2784324b77b272e40f57d6b" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:34.032336 9343 raft_consensus.cc:2454] T 8482be0fb2784324b77b272e40f57d6b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:34.032490 9016 leader_election.cc:302] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:34.032670 10527 raft_consensus.cc:2781] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:34.032742 10527 raft_consensus.cc:482] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:34.032795 10527 raft_consensus.cc:3037] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.033835 10527 raft_consensus.cc:504] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:34.034170 10527 leader_election.cc:288] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:34.034505 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8482be0fb2784324b77b272e40f57d6b" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:34.034552 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8482be0fb2784324b77b272e40f57d6b" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:34.034623 8947 raft_consensus.cc:3037] T 8482be0fb2784324b77b272e40f57d6b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.034673 9343 raft_consensus.cc:3037] T 8482be0fb2784324b77b272e40f57d6b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.035423 8947 raft_consensus.cc:2454] T 8482be0fb2784324b77b272e40f57d6b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:34.035614 9343 raft_consensus.cc:2454] T 8482be0fb2784324b77b272e40f57d6b P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:34.035840 9016 leader_election.cc:302] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:34.036012 10527 raft_consensus.cc:2781] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:34.036096 10527 raft_consensus.cc:686] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 13:59:34.036238 10527 consensus_queue.cc:237] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:34.038587 6386 catalog_manager.cc:5484] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:34.057590 10535 raft_consensus.cc:482] T ee295fdb8a2e4990a1e0b47dc30cd56b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:34.057722 10535 raft_consensus.cc:504] T ee295fdb8a2e4990a1e0b47dc30cd56b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.058146 10535 leader_election.cc:288] T ee295fdb8a2e4990a1e0b47dc30cd56b P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:34.058552 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee295fdb8a2e4990a1e0b47dc30cd56b" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:34.058702 9205 raft_consensus.cc:2454] T ee295fdb8a2e4990a1e0b47dc30cd56b P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:34.058972 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee295fdb8a2e4990a1e0b47dc30cd56b" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:34.059072 8883 leader_election.cc:302] T ee295fdb8a2e4990a1e0b47dc30cd56b P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:34.059099 6920 raft_consensus.cc:2454] T ee295fdb8a2e4990a1e0b47dc30cd56b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:34.059253 10535 raft_consensus.cc:2781] T ee295fdb8a2e4990a1e0b47dc30cd56b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:34.059355 10535 raft_consensus.cc:482] T ee295fdb8a2e4990a1e0b47dc30cd56b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:34.059410 10535 raft_consensus.cc:3037] T ee295fdb8a2e4990a1e0b47dc30cd56b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.060349 10535 raft_consensus.cc:504] T ee295fdb8a2e4990a1e0b47dc30cd56b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.060657 10535 leader_election.cc:288] T ee295fdb8a2e4990a1e0b47dc30cd56b P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:34.060976 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee295fdb8a2e4990a1e0b47dc30cd56b" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:34.061085 9205 raft_consensus.cc:3037] T ee295fdb8a2e4990a1e0b47dc30cd56b P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.061095 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee295fdb8a2e4990a1e0b47dc30cd56b" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:34.061233 6920 raft_consensus.cc:3037] T ee295fdb8a2e4990a1e0b47dc30cd56b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.061997 9205 raft_consensus.cc:2454] T ee295fdb8a2e4990a1e0b47dc30cd56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:34.062290 6920 raft_consensus.cc:2454] T ee295fdb8a2e4990a1e0b47dc30cd56b P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:34.062541 8883 leader_election.cc:302] T ee295fdb8a2e4990a1e0b47dc30cd56b P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:34.062819 10535 raft_consensus.cc:2781] T ee295fdb8a2e4990a1e0b47dc30cd56b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:34.062903 10535 raft_consensus.cc:686] T ee295fdb8a2e4990a1e0b47dc30cd56b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 13:59:34.063055 10535 consensus_queue.cc:237] T ee295fdb8a2e4990a1e0b47dc30cd56b P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.065728 6379 catalog_manager.cc:5484] T ee295fdb8a2e4990a1e0b47dc30cd56b P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:34.168573 10304 raft_consensus.cc:482] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:34.168704 10304 raft_consensus.cc:504] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:34.169097 10304 leader_election.cc:288] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:34.169472 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ad9e6c93b046338e5b0003821e9803" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:34.169611 8947 raft_consensus.cc:2454] T d5ad9e6c93b046338e5b0003821e9803 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:34.169636 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ad9e6c93b046338e5b0003821e9803" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:34.169799 9081 raft_consensus.cc:2454] T d5ad9e6c93b046338e5b0003821e9803 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:34.169939 9147 leader_election.cc:302] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:34.170115 10304 raft_consensus.cc:2781] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:34.170181 10304 raft_consensus.cc:482] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:34.170240 10304 raft_consensus.cc:3037] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.171126 10304 raft_consensus.cc:504] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:34.171499 10304 leader_election.cc:288] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Requested vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:34.171836 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ad9e6c93b046338e5b0003821e9803" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:34.171954 8947 raft_consensus.cc:3037] T d5ad9e6c93b046338e5b0003821e9803 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.171931 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ad9e6c93b046338e5b0003821e9803" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:34.172065 9081 raft_consensus.cc:3037] T d5ad9e6c93b046338e5b0003821e9803 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.172758 8947 raft_consensus.cc:2454] T d5ad9e6c93b046338e5b0003821e9803 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:34.172857 9081 raft_consensus.cc:2454] T d5ad9e6c93b046338e5b0003821e9803 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:34.173035 9147 leader_election.cc:302] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:34.173235 10304 raft_consensus.cc:2781] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:34.173323 10304 raft_consensus.cc:686] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 13:59:34.173460 10304 consensus_queue.cc:237] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:34.175964 6379 catalog_manager.cc:5484] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:34.235814 10535 raft_consensus.cc:482] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:34.235941 10535 raft_consensus.cc:504] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:34.236312 10535 leader_election.cc:288] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:34.236660 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0daf41253f4678b158e3ccbb7b0c98" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:34.236788 9205 raft_consensus.cc:2454] T 9f0daf41253f4678b158e3ccbb7b0c98 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:34.237011 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0daf41253f4678b158e3ccbb7b0c98" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:34.237061 8883 leader_election.cc:302] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:34.237143 9343 raft_consensus.cc:2454] T 9f0daf41253f4678b158e3ccbb7b0c98 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:34.237334 10535 raft_consensus.cc:2781] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:34.237419 10535 raft_consensus.cc:482] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:34.237471 10535 raft_consensus.cc:3037] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.238611 10535 raft_consensus.cc:504] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:34.239112 10535 leader_election.cc:288] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:34.239323 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0daf41253f4678b158e3ccbb7b0c98" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:34.239463 9205 raft_consensus.cc:3037] T 9f0daf41253f4678b158e3ccbb7b0c98 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.239529 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0daf41253f4678b158e3ccbb7b0c98" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:34.239639 9343 raft_consensus.cc:3037] T 9f0daf41253f4678b158e3ccbb7b0c98 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.240374 9205 raft_consensus.cc:2454] T 9f0daf41253f4678b158e3ccbb7b0c98 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:34.240458 9343 raft_consensus.cc:2454] T 9f0daf41253f4678b158e3ccbb7b0c98 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:34.240684 8883 leader_election.cc:302] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:34.240834 10535 raft_consensus.cc:2781] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:34.240932 10535 raft_consensus.cc:686] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 13:59:34.241096 10535 consensus_queue.cc:237] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:34.243752 6379 catalog_manager.cc:5484] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:34.300743 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:34.304817 8947 raft_consensus.cc:1261] T d5ad9e6c93b046338e5b0003821e9803 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:34.305357 10304 consensus_queue.cc:1035] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:34.307906 9081 raft_consensus.cc:1261] T d5ad9e6c93b046338e5b0003821e9803 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:34.308426 6379 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet d5ad9e6c93b046338e5b0003821e9803 (table rebalancer_dynamic_table [id=e85ab5044d6344f9ad4df66b7cdddcef]). Expected version 1 got 0
I20240503 13:59:34.308434 10443 consensus_queue.cc:1035] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:34.310142 6379 catalog_manager.cc:5576] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091) does not have the latest schema for tablet d5ad9e6c93b046338e5b0003821e9803 (table rebalancer_dynamic_table [id=e85ab5044d6344f9ad4df66b7cdddcef]). Expected version 1 got 0
I20240503 13:59:34.313859 10544 tablet.cc:1722] T d5ad9e6c93b046338e5b0003821e9803 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:34.314391 6379 catalog_manager.cc:5576] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481) does not have the latest schema for tablet d5ad9e6c93b046338e5b0003821e9803 (table rebalancer_dynamic_table [id=e85ab5044d6344f9ad4df66b7cdddcef]). Expected version 1 got 0
I20240503 13:59:34.314800 10544 tablet.cc:1620] T d5ad9e6c93b046338e5b0003821e9803 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:34.316062 6386 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet d5ad9e6c93b046338e5b0003821e9803 (table rebalancer_dynamic_table [id=e85ab5044d6344f9ad4df66b7cdddcef]). Expected version 1 got 0
I20240503 13:59:34.320557 6386 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=e85ab5044d6344f9ad4df66b7cdddcef] alter complete (version 1)
I20240503 13:59:34.323503 10540 tablet.cc:1722] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:34.323791 10540 tablet.cc:1620] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:34.324555 10547 tablet.cc:1722] T d5ad9e6c93b046338e5b0003821e9803 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:34.324868 10547 tablet.cc:1620] T d5ad9e6c93b046338e5b0003821e9803 P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:34.422106 6386 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:34.426985 6386 catalog_manager.cc:5576] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091) does not have the latest schema for tablet d5ad9e6c93b046338e5b0003821e9803 (table rebalancer_dynamic_table [id=e85ab5044d6344f9ad4df66b7cdddcef]). Expected version 2 got 1
I20240503 13:59:34.427554 10540 tablet.cc:1722] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:34.427865 10540 tablet.cc:1620] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:34.428390 10544 tablet.cc:1722] T d5ad9e6c93b046338e5b0003821e9803 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:34.428593 10547 tablet.cc:1722] T d5ad9e6c93b046338e5b0003821e9803 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:34.428779 10544 tablet.cc:1620] T d5ad9e6c93b046338e5b0003821e9803 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:34.428893 10547 tablet.cc:1620] T d5ad9e6c93b046338e5b0003821e9803 P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:34.433238 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=e85ab5044d6344f9ad4df66b7cdddcef] alter complete (version 2)
I20240503 13:59:34.478279 10546 consensus_queue.cc:1035] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:34.481334 10527 consensus_queue.cc:1035] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:34.535542 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:34.538997 10535 consensus_queue.cc:1035] T ee295fdb8a2e4990a1e0b47dc30cd56b P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:34.541608 10540 tablet.cc:1722] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:34.543681 10547 tablet.cc:1722] T d5ad9e6c93b046338e5b0003821e9803 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:34.544656 10534 consensus_queue.cc:1035] T ee295fdb8a2e4990a1e0b47dc30cd56b P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:34.549225 10544 tablet.cc:1722] T d5ad9e6c93b046338e5b0003821e9803 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:34.549324 6386 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=e85ab5044d6344f9ad4df66b7cdddcef] alter complete (version 3)
I20240503 13:59:34.652354 6379 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:34.652529 6379 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:34.654614 6379 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet d5ad9e6c93b046338e5b0003821e9803
I20240503 13:59:34.655088 8928 tablet_service.cc:1514] Processing DeleteTablet for tablet d5ad9e6c93b046338e5b0003821e9803 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:34 UTC) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:34.655213 9190 tablet_service.cc:1514] Processing DeleteTablet for tablet d5ad9e6c93b046338e5b0003821e9803 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:34 UTC) from {username='slave'} at 127.0.0.1:57960
I20240503 13:59:34.655457 10555 tablet_replica.cc:330] T d5ad9e6c93b046338e5b0003821e9803 P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:34.655434 9058 tablet_service.cc:1514] Processing DeleteTablet for tablet d5ad9e6c93b046338e5b0003821e9803 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:34 UTC) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:34.655535 10556 tablet_replica.cc:330] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 13:59:34.655592 10555 raft_consensus.cc:2229] T d5ad9e6c93b046338e5b0003821e9803 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:34.655666 10556 raft_consensus.cc:2229] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:34.655738 10555 raft_consensus.cc:2258] T d5ad9e6c93b046338e5b0003821e9803 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:34.655753 10557 tablet_replica.cc:330] T d5ad9e6c93b046338e5b0003821e9803 P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:34.655875 10557 raft_consensus.cc:2229] T d5ad9e6c93b046338e5b0003821e9803 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:34.655900 10556 raft_consensus.cc:2258] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:34.656005 10557 raft_consensus.cc:2258] T d5ad9e6c93b046338e5b0003821e9803 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:34.656402 10555 ts_tablet_manager.cc:1875] T d5ad9e6c93b046338e5b0003821e9803 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:34.656466 10556 ts_tablet_manager.cc:1875] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:34.657169 10557 ts_tablet_manager.cc:1875] T d5ad9e6c93b046338e5b0003821e9803 P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:34.659262 10555 ts_tablet_manager.cc:1888] T d5ad9e6c93b046338e5b0003821e9803 P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:34.659360 10555 log.cc:1196] T d5ad9e6c93b046338e5b0003821e9803 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/d5ad9e6c93b046338e5b0003821e9803
I20240503 13:59:34.659727 10555 ts_tablet_manager.cc:1909] T d5ad9e6c93b046338e5b0003821e9803 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting consensus metadata
I20240503 13:59:34.659761 10557 ts_tablet_manager.cc:1888] T d5ad9e6c93b046338e5b0003821e9803 P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:34.659879 10557 log.cc:1196] T d5ad9e6c93b046338e5b0003821e9803 P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/d5ad9e6c93b046338e5b0003821e9803
I20240503 13:59:34.660035 10556 ts_tablet_manager.cc:1888] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:34.660125 10556 log.cc:1196] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/d5ad9e6c93b046338e5b0003821e9803
I20240503 13:59:34.660216 10557 ts_tablet_manager.cc:1909] T d5ad9e6c93b046338e5b0003821e9803 P 8d574fb4a89d4eea947c859470d44dd4: Deleting consensus metadata
I20240503 13:59:34.660400 10556 ts_tablet_manager.cc:1909] T d5ad9e6c93b046338e5b0003821e9803 P ab1d21935d6049888eb0ceccc3c9d735: Deleting consensus metadata
I20240503 13:59:34.660550 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet d5ad9e6c93b046338e5b0003821e9803 (table rebalancer_dynamic_table_new_name [id=e85ab5044d6344f9ad4df66b7cdddcef]) successfully deleted
I20240503 13:59:34.660995 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet d5ad9e6c93b046338e5b0003821e9803 (table rebalancer_dynamic_table_new_name [id=e85ab5044d6344f9ad4df66b7cdddcef]) successfully deleted
I20240503 13:59:34.661291 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet d5ad9e6c93b046338e5b0003821e9803 (table rebalancer_dynamic_table_new_name [id=e85ab5044d6344f9ad4df66b7cdddcef]) successfully deleted
I20240503 13:59:34.756337 6379 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:34.759218 10552 consensus_queue.cc:1035] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:34.762216 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 60eef9a26be04a579a4831488c958606 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=daa36562f64a42fc8950d4fa0cb21dfe]), partition=
I20240503 13:59:34.762213 9322 tablet_service.cc:1467] Processing CreateTablet for tablet 60eef9a26be04a579a4831488c958606 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=daa36562f64a42fc8950d4fa0cb21dfe]), partition=
I20240503 13:59:34.762488 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60eef9a26be04a579a4831488c958606. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.762488 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60eef9a26be04a579a4831488c958606. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.762650 9190 tablet_service.cc:1467] Processing CreateTablet for tablet 60eef9a26be04a579a4831488c958606 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=daa36562f64a42fc8950d4fa0cb21dfe]), partition=
I20240503 13:59:34.762850 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60eef9a26be04a579a4831488c958606. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.765115 10561 tablet_bootstrap.cc:492] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:34.766223 10561 tablet_bootstrap.cc:654] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.766880 10560 tablet_bootstrap.cc:492] T 60eef9a26be04a579a4831488c958606 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:34.766897 10534 consensus_queue.cc:1035] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:34.768159 10560 tablet_bootstrap.cc:654] T 60eef9a26be04a579a4831488c958606 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.769795 10559 tablet_bootstrap.cc:492] T 60eef9a26be04a579a4831488c958606 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:34.770941 10559 tablet_bootstrap.cc:654] T 60eef9a26be04a579a4831488c958606 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.771999 10561 tablet_bootstrap.cc:492] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:34.772159 10561 ts_tablet_manager.cc:1367] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:59:34.772274 10560 tablet_bootstrap.cc:492] T 60eef9a26be04a579a4831488c958606 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:34.772338 10559 tablet_bootstrap.cc:492] T 60eef9a26be04a579a4831488c958606 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:34.772382 10560 ts_tablet_manager.cc:1367] T 60eef9a26be04a579a4831488c958606 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 13:59:34.772416 10559 ts_tablet_manager.cc:1367] T 60eef9a26be04a579a4831488c958606 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:59:34.772697 10561 raft_consensus.cc:348] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.772840 10561 raft_consensus.cc:374] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.772884 10561 raft_consensus.cc:729] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.772886 10559 raft_consensus.cc:348] T 60eef9a26be04a579a4831488c958606 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.773016 10559 raft_consensus.cc:374] T 60eef9a26be04a579a4831488c958606 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.772924 10560 raft_consensus.cc:348] T 60eef9a26be04a579a4831488c958606 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.773077 10559 raft_consensus.cc:729] T 60eef9a26be04a579a4831488c958606 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.773114 10560 raft_consensus.cc:374] T 60eef9a26be04a579a4831488c958606 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.773082 10561 consensus_queue.cc:260] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.773202 10560 raft_consensus.cc:729] T 60eef9a26be04a579a4831488c958606 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.773273 10559 consensus_queue.cc:260] T 60eef9a26be04a579a4831488c958606 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.773344 10560 consensus_queue.cc:260] T 60eef9a26be04a579a4831488c958606 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.773568 10561 ts_tablet_manager.cc:1398] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:34.774329 10560 ts_tablet_manager.cc:1398] T 60eef9a26be04a579a4831488c958606 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:34.775063 10559 ts_tablet_manager.cc:1398] T 60eef9a26be04a579a4831488c958606 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:34.782940 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:34.789168 9190 tablet_service.cc:1467] Processing CreateTablet for tablet 1eec99c0bb994dd7b6ae63918125bccb (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=800413c4c03946b7bfef618ffc28c99e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.789489 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eec99c0bb994dd7b6ae63918125bccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.790009 9323 tablet_service.cc:1467] Processing CreateTablet for tablet 9af92a2d926543c4aa68a1f85b7da24b (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=800413c4c03946b7bfef618ffc28c99e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.790309 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9af92a2d926543c4aa68a1f85b7da24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.790024 9322 tablet_service.cc:1467] Processing CreateTablet for tablet 1eec99c0bb994dd7b6ae63918125bccb (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=800413c4c03946b7bfef618ffc28c99e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.790683 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eec99c0bb994dd7b6ae63918125bccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.791893 10560 tablet_bootstrap.cc:492] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:34.792707 6508 tablet_service.cc:1467] Processing CreateTablet for tablet bea48553d8c546abbb58f4d488e6b09a (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=800413c4c03946b7bfef618ffc28c99e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.792906 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 1eec99c0bb994dd7b6ae63918125bccb (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=800413c4c03946b7bfef618ffc28c99e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.793015 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bea48553d8c546abbb58f4d488e6b09a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.792986 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 9af92a2d926543c4aa68a1f85b7da24b (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=800413c4c03946b7bfef618ffc28c99e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.793170 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eec99c0bb994dd7b6ae63918125bccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.793218 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9af92a2d926543c4aa68a1f85b7da24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.793823 10560 tablet_bootstrap.cc:654] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.795231 10559 tablet_bootstrap.cc:492] T 1eec99c0bb994dd7b6ae63918125bccb P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:34.795576 8926 tablet_service.cc:1467] Processing CreateTablet for tablet 9af92a2d926543c4aa68a1f85b7da24b (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=800413c4c03946b7bfef618ffc28c99e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.795765 8926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9af92a2d926543c4aa68a1f85b7da24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.795934 10569 tablet_bootstrap.cc:492] T bea48553d8c546abbb58f4d488e6b09a P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:34.796933 10569 tablet_bootstrap.cc:654] T bea48553d8c546abbb58f4d488e6b09a P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.797075 9061 tablet_service.cc:1467] Processing CreateTablet for tablet bea48553d8c546abbb58f4d488e6b09a (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=800413c4c03946b7bfef618ffc28c99e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.797317 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bea48553d8c546abbb58f4d488e6b09a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.797971 10568 tablet_bootstrap.cc:492] T 9af92a2d926543c4aa68a1f85b7da24b P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:34.798194 10569 tablet_bootstrap.cc:492] T bea48553d8c546abbb58f4d488e6b09a P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:34.798272 10569 ts_tablet_manager.cc:1367] T bea48553d8c546abbb58f4d488e6b09a P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:34.798759 10569 raft_consensus.cc:348] T bea48553d8c546abbb58f4d488e6b09a P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.798869 10569 raft_consensus.cc:374] T bea48553d8c546abbb58f4d488e6b09a P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.798911 10569 raft_consensus.cc:729] T bea48553d8c546abbb58f4d488e6b09a P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.799046 10569 consensus_queue.cc:260] T bea48553d8c546abbb58f4d488e6b09a P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.799151 10568 tablet_bootstrap.cc:654] T 9af92a2d926543c4aa68a1f85b7da24b P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.799360 10569 ts_tablet_manager.cc:1398] T bea48553d8c546abbb58f4d488e6b09a P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:34.799942 8926 tablet_service.cc:1467] Processing CreateTablet for tablet bea48553d8c546abbb58f4d488e6b09a (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=800413c4c03946b7bfef618ffc28c99e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.800186 8926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bea48553d8c546abbb58f4d488e6b09a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.801925 10559 tablet_bootstrap.cc:654] T 1eec99c0bb994dd7b6ae63918125bccb P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.802554 10561 tablet_bootstrap.cc:492] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:34.803016 10559 tablet_bootstrap.cc:492] T 1eec99c0bb994dd7b6ae63918125bccb P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:34.803172 10559 ts_tablet_manager.cc:1367] T 1eec99c0bb994dd7b6ae63918125bccb P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:59:34.803628 10561 tablet_bootstrap.cc:654] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.803814 10559 raft_consensus.cc:348] T 1eec99c0bb994dd7b6ae63918125bccb P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:34.803921 10559 raft_consensus.cc:374] T 1eec99c0bb994dd7b6ae63918125bccb P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.803943 10559 raft_consensus.cc:729] T 1eec99c0bb994dd7b6ae63918125bccb P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.804064 10559 consensus_queue.cc:260] T 1eec99c0bb994dd7b6ae63918125bccb P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:34.804526 10559 ts_tablet_manager.cc:1398] T 1eec99c0bb994dd7b6ae63918125bccb P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:34.804708 10559 tablet_bootstrap.cc:492] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:34.804894 10561 tablet_bootstrap.cc:492] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:34.804991 10561 ts_tablet_manager.cc:1367] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:34.805505 10561 raft_consensus.cc:348] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:34.805586 10559 tablet_bootstrap.cc:654] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.805616 10568 tablet_bootstrap.cc:492] T 9af92a2d926543c4aa68a1f85b7da24b P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:34.805613 10561 raft_consensus.cc:374] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.805651 10561 raft_consensus.cc:729] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.805701 10568 ts_tablet_manager.cc:1367] T 9af92a2d926543c4aa68a1f85b7da24b P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240503 13:59:34.805811 10560 tablet_bootstrap.cc:492] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:34.805794 10561 consensus_queue.cc:260] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:34.805939 10560 ts_tablet_manager.cc:1367] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.003s
I20240503 13:59:34.806136 10561 ts_tablet_manager.cc:1398] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:34.806291 10561 tablet_bootstrap.cc:492] T 9af92a2d926543c4aa68a1f85b7da24b P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:34.806244 10568 raft_consensus.cc:348] T 9af92a2d926543c4aa68a1f85b7da24b P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.806368 10568 raft_consensus.cc:374] T 9af92a2d926543c4aa68a1f85b7da24b P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.806403 10568 raft_consensus.cc:729] T 9af92a2d926543c4aa68a1f85b7da24b P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.806428 10560 raft_consensus.cc:348] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:34.806527 10560 raft_consensus.cc:374] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.806558 10560 raft_consensus.cc:729] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.806529 10568 consensus_queue.cc:260] T 9af92a2d926543c4aa68a1f85b7da24b P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.806717 10560 consensus_queue.cc:260] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:34.806813 10568 ts_tablet_manager.cc:1398] T 9af92a2d926543c4aa68a1f85b7da24b P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:34.806974 10568 tablet_bootstrap.cc:492] T bea48553d8c546abbb58f4d488e6b09a P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:34.807097 10560 ts_tablet_manager.cc:1398] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:34.807368 10561 tablet_bootstrap.cc:654] T 9af92a2d926543c4aa68a1f85b7da24b P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.807843 10568 tablet_bootstrap.cc:654] T bea48553d8c546abbb58f4d488e6b09a P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.808449 10561 tablet_bootstrap.cc:492] T 9af92a2d926543c4aa68a1f85b7da24b P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:34.808532 10561 ts_tablet_manager.cc:1367] T 9af92a2d926543c4aa68a1f85b7da24b P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:34.808985 10559 tablet_bootstrap.cc:492] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:34.808992 10561 raft_consensus.cc:348] T 9af92a2d926543c4aa68a1f85b7da24b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.809072 10559 ts_tablet_manager.cc:1367] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240503 13:59:34.809085 10561 raft_consensus.cc:374] T 9af92a2d926543c4aa68a1f85b7da24b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.809118 10561 raft_consensus.cc:729] T 9af92a2d926543c4aa68a1f85b7da24b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.809230 10561 consensus_queue.cc:260] T 9af92a2d926543c4aa68a1f85b7da24b P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.809509 10561 ts_tablet_manager.cc:1398] T 9af92a2d926543c4aa68a1f85b7da24b P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:34.809477 10559 raft_consensus.cc:348] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.809553 10559 raft_consensus.cc:374] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.809587 10559 raft_consensus.cc:729] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.809713 10559 consensus_queue.cc:260] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.810200 10559 ts_tablet_manager.cc:1398] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:34.810389 10559 tablet_bootstrap.cc:492] T bea48553d8c546abbb58f4d488e6b09a P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:34.811439 10559 tablet_bootstrap.cc:654] T bea48553d8c546abbb58f4d488e6b09a P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.816165 10568 tablet_bootstrap.cc:492] T bea48553d8c546abbb58f4d488e6b09a P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:34.816264 10568 ts_tablet_manager.cc:1367] T bea48553d8c546abbb58f4d488e6b09a P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:59:34.817997 10568 raft_consensus.cc:348] T bea48553d8c546abbb58f4d488e6b09a P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.818130 10568 raft_consensus.cc:374] T bea48553d8c546abbb58f4d488e6b09a P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.818161 10568 raft_consensus.cc:729] T bea48553d8c546abbb58f4d488e6b09a P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.818276 10568 consensus_queue.cc:260] T bea48553d8c546abbb58f4d488e6b09a P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.818539 10568 ts_tablet_manager.cc:1398] T bea48553d8c546abbb58f4d488e6b09a P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:34.819583 10559 tablet_bootstrap.cc:492] T bea48553d8c546abbb58f4d488e6b09a P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:34.819674 10559 ts_tablet_manager.cc:1367] T bea48553d8c546abbb58f4d488e6b09a P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:59:34.820188 10559 raft_consensus.cc:348] T bea48553d8c546abbb58f4d488e6b09a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.820287 10559 raft_consensus.cc:374] T bea48553d8c546abbb58f4d488e6b09a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.820318 10559 raft_consensus.cc:729] T bea48553d8c546abbb58f4d488e6b09a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.820449 10559 consensus_queue.cc:260] T bea48553d8c546abbb58f4d488e6b09a P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.823298 10559 ts_tablet_manager.cc:1398] T bea48553d8c546abbb58f4d488e6b09a P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:59:34.831926 10564 raft_consensus.cc:482] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:34.832039 10564 raft_consensus.cc:504] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.832531 10564 leader_election.cc:288] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:34.832927 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60eef9a26be04a579a4831488c958606" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:34.832923 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60eef9a26be04a579a4831488c958606" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:34.833056 9211 raft_consensus.cc:2454] T 60eef9a26be04a579a4831488c958606 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:34.833056 8947 raft_consensus.cc:2454] T 60eef9a26be04a579a4831488c958606 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:34.833354 9278 leader_election.cc:302] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:34.833542 10564 raft_consensus.cc:2781] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:34.833602 10564 raft_consensus.cc:482] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:34.833657 10564 raft_consensus.cc:3037] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.834558 10564 raft_consensus.cc:504] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.834893 10564 leader_election.cc:288] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:34.835300 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60eef9a26be04a579a4831488c958606" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:34.835333 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60eef9a26be04a579a4831488c958606" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:34.835418 8947 raft_consensus.cc:3037] T 60eef9a26be04a579a4831488c958606 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.835429 9211 raft_consensus.cc:3037] T 60eef9a26be04a579a4831488c958606 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.836344 8947 raft_consensus.cc:2454] T 60eef9a26be04a579a4831488c958606 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:34.836344 9211 raft_consensus.cc:2454] T 60eef9a26be04a579a4831488c958606 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:34.836616 9278 leader_election.cc:302] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 0d77752181e944698e809e325e5937e6; no voters:
I20240503 13:59:34.836786 10339 raft_consensus.cc:2781] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:34.836869 10339 raft_consensus.cc:686] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:34.837002 10339 consensus_queue.cc:237] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.839167 6380 catalog_manager.cc:5484] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:34.922487 10552 raft_consensus.cc:482] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:34.922618 10552 raft_consensus.cc:504] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.923017 10552 leader_election.cc:288] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:34.923271 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af92a2d926543c4aa68a1f85b7da24b" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:34.923424 9081 raft_consensus.cc:2454] T 9af92a2d926543c4aa68a1f85b7da24b P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:34.923539 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af92a2d926543c4aa68a1f85b7da24b" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:34.923671 9343 raft_consensus.cc:2454] T 9af92a2d926543c4aa68a1f85b7da24b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:34.923681 8883 leader_election.cc:302] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:34.923844 10552 raft_consensus.cc:2781] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:34.923900 10552 raft_consensus.cc:482] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:34.923945 10552 raft_consensus.cc:3037] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.924981 10552 raft_consensus.cc:504] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.925323 10552 leader_election.cc:288] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:34.925591 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af92a2d926543c4aa68a1f85b7da24b" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:34.925643 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af92a2d926543c4aa68a1f85b7da24b" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:34.925709 9081 raft_consensus.cc:3037] T 9af92a2d926543c4aa68a1f85b7da24b P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.925736 9343 raft_consensus.cc:3037] T 9af92a2d926543c4aa68a1f85b7da24b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.926643 9343 raft_consensus.cc:2454] T 9af92a2d926543c4aa68a1f85b7da24b P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:34.926643 9081 raft_consensus.cc:2454] T 9af92a2d926543c4aa68a1f85b7da24b P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:34.926941 8884 leader_election.cc:302] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 0d77752181e944698e809e325e5937e6; no voters:
I20240503 13:59:34.927449 10552 raft_consensus.cc:2781] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:34.927568 10552 raft_consensus.cc:686] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 13:59:34.928059 10552 consensus_queue.cc:237] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.929879 6380 catalog_manager.cc:5484] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:34.947008 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:34.950742 8947 raft_consensus.cc:1261] T 60eef9a26be04a579a4831488c958606 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:34.951238 10339 consensus_queue.cc:1035] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:34.951416 9211 raft_consensus.cc:1261] T 60eef9a26be04a579a4831488c958606 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:34.951784 10339 consensus_queue.cc:1035] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:34.953734 10582 tablet.cc:1722] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:34.953809 10550 mvcc.cc:205] Tried to move back new op lower bound from 7023594598194569216 to 7023594597733855232. Current Snapshot: MvccSnapshot[applied={T|T < 7023594598194569216}]
I20240503 13:59:34.954061 10582 tablet.cc:1620] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:34.954151 6380 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet 60eef9a26be04a579a4831488c958606 (table rebalancer_dynamic_table [id=daa36562f64a42fc8950d4fa0cb21dfe]). Expected version 1 got 0
I20240503 13:59:34.954717 6379 catalog_manager.cc:5576] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091) does not have the latest schema for tablet 60eef9a26be04a579a4831488c958606 (table rebalancer_dynamic_table [id=daa36562f64a42fc8950d4fa0cb21dfe]). Expected version 1 got 0
I20240503 13:59:34.956014 10540 tablet.cc:1722] T 60eef9a26be04a579a4831488c958606 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:34.956300 10540 tablet.cc:1620] T 60eef9a26be04a579a4831488c958606 P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:34.956867 6379 catalog_manager.cc:5576] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091) does not have the latest schema for tablet 60eef9a26be04a579a4831488c958606 (table rebalancer_dynamic_table [id=daa36562f64a42fc8950d4fa0cb21dfe]). Expected version 1 got 0
I20240503 13:59:34.957662 10544 tablet.cc:1722] T 60eef9a26be04a579a4831488c958606 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:34.957978 10544 tablet.cc:1620] T 60eef9a26be04a579a4831488c958606 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:34.959733 10552 raft_consensus.cc:482] T bea48553d8c546abbb58f4d488e6b09a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:34.959817 10552 raft_consensus.cc:504] T bea48553d8c546abbb58f4d488e6b09a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.960212 10552 leader_election.cc:288] T bea48553d8c546abbb58f4d488e6b09a P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:34.961604 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bea48553d8c546abbb58f4d488e6b09a" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:34.961766 6528 raft_consensus.cc:2454] T bea48553d8c546abbb58f4d488e6b09a P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:34.962062 8883 leader_election.cc:302] T bea48553d8c546abbb58f4d488e6b09a P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:34.962278 10552 raft_consensus.cc:2781] T bea48553d8c546abbb58f4d488e6b09a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:34.962347 10552 raft_consensus.cc:482] T bea48553d8c546abbb58f4d488e6b09a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:34.962399 10552 raft_consensus.cc:3037] T bea48553d8c546abbb58f4d488e6b09a P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.962698 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=daa36562f64a42fc8950d4fa0cb21dfe] alter complete (version 1)
I20240503 13:59:34.963325 10552 raft_consensus.cc:504] T bea48553d8c546abbb58f4d488e6b09a P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.963496 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bea48553d8c546abbb58f4d488e6b09a" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:34.963618 9081 raft_consensus.cc:2454] T bea48553d8c546abbb58f4d488e6b09a P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:34.963629 10552 leader_election.cc:288] T bea48553d8c546abbb58f4d488e6b09a P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:34.964016 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bea48553d8c546abbb58f4d488e6b09a" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:34.964123 6528 raft_consensus.cc:3037] T bea48553d8c546abbb58f4d488e6b09a P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.965111 6528 raft_consensus.cc:2454] T bea48553d8c546abbb58f4d488e6b09a P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:34.965425 8883 leader_election.cc:302] T bea48553d8c546abbb58f4d488e6b09a P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:34.965613 10552 raft_consensus.cc:2781] T bea48553d8c546abbb58f4d488e6b09a P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:34.965698 10552 raft_consensus.cc:686] T bea48553d8c546abbb58f4d488e6b09a P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 13:59:34.965840 10552 consensus_queue.cc:237] T bea48553d8c546abbb58f4d488e6b09a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:34.967062 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bea48553d8c546abbb58f4d488e6b09a" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:34.967229 9081 raft_consensus.cc:3037] T bea48553d8c546abbb58f4d488e6b09a P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.967657 6386 catalog_manager.cc:5484] T bea48553d8c546abbb58f4d488e6b09a P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:34.968075 9081 raft_consensus.cc:2454] T bea48553d8c546abbb58f4d488e6b09a P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:35.066176 6386 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:35.071444 10582 tablet.cc:1722] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:35.071784 10582 tablet.cc:1620] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:35.071884 10540 tablet.cc:1722] T 60eef9a26be04a579a4831488c958606 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:35.072186 10540 tablet.cc:1620] T 60eef9a26be04a579a4831488c958606 P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:35.072549 10544 tablet.cc:1722] T 60eef9a26be04a579a4831488c958606 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:35.072813 10544 tablet.cc:1620] T 60eef9a26be04a579a4831488c958606 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:35.077332 6383 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=daa36562f64a42fc8950d4fa0cb21dfe] alter complete (version 2)
I20240503 13:59:35.092216 10583 raft_consensus.cc:482] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:35.092321 10583 raft_consensus.cc:504] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:35.092686 10583 leader_election.cc:288] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:35.093004 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eec99c0bb994dd7b6ae63918125bccb" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:35.093070 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eec99c0bb994dd7b6ae63918125bccb" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:35.093153 9211 raft_consensus.cc:2454] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:35.093200 8947 raft_consensus.cc:2454] T 1eec99c0bb994dd7b6ae63918125bccb P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:35.093459 9278 leader_election.cc:302] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 0d77752181e944698e809e325e5937e6; no voters:
I20240503 13:59:35.093631 10583 raft_consensus.cc:2781] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:35.093689 10583 raft_consensus.cc:482] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:35.093736 10583 raft_consensus.cc:3037] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.094650 10583 raft_consensus.cc:504] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:35.094954 10583 leader_election.cc:288] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:35.095237 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eec99c0bb994dd7b6ae63918125bccb" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:35.095387 9211 raft_consensus.cc:3037] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.095433 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eec99c0bb994dd7b6ae63918125bccb" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:35.095556 8947 raft_consensus.cc:3037] T 1eec99c0bb994dd7b6ae63918125bccb P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.096239 9211 raft_consensus.cc:2454] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:35.096495 9278 leader_election.cc:302] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:35.096509 8947 raft_consensus.cc:2454] T 1eec99c0bb994dd7b6ae63918125bccb P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:35.096670 10583 raft_consensus.cc:2781] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:35.096756 10583 raft_consensus.cc:686] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:35.096946 10583 consensus_queue.cc:237] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:35.099000 6386 catalog_manager.cc:5484] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:35.181053 6386 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:35.186009 10582 tablet.cc:1722] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:35.186410 10540 tablet.cc:1722] T 60eef9a26be04a579a4831488c958606 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:35.186425 10544 tablet.cc:1722] T 60eef9a26be04a579a4831488c958606 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:35.191401 6383 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=daa36562f64a42fc8950d4fa0cb21dfe] alter complete (version 3)
I20240503 13:59:35.294021 6383 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:35.294190 6383 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:35.296171 6383 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 60eef9a26be04a579a4831488c958606
I20240503 13:59:35.296772 9190 tablet_service.cc:1514] Processing DeleteTablet for tablet 60eef9a26be04a579a4831488c958606 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:35 UTC) from {username='slave'} at 127.0.0.1:57960
I20240503 13:59:35.296852 9322 tablet_service.cc:1514] Processing DeleteTablet for tablet 60eef9a26be04a579a4831488c958606 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:35 UTC) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:35.296837 8926 tablet_service.cc:1514] Processing DeleteTablet for tablet 60eef9a26be04a579a4831488c958606 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:35 UTC) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:35.297144 10589 tablet_replica.cc:330] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:35.297140 10590 tablet_replica.cc:330] T 60eef9a26be04a579a4831488c958606 P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:35.297276 10588 tablet_replica.cc:330] T 60eef9a26be04a579a4831488c958606 P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 13:59:35.297309 10590 raft_consensus.cc:2229] T 60eef9a26be04a579a4831488c958606 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:35.297353 10589 raft_consensus.cc:2229] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:35.297436 10590 raft_consensus.cc:2258] T 60eef9a26be04a579a4831488c958606 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:35.297433 10588 raft_consensus.cc:2229] T 60eef9a26be04a579a4831488c958606 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:35.297564 10588 raft_consensus.cc:2258] T 60eef9a26be04a579a4831488c958606 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:35.297585 10589 raft_consensus.cc:2258] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:35.298094 10590 ts_tablet_manager.cc:1875] T 60eef9a26be04a579a4831488c958606 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:35.298177 10588 ts_tablet_manager.cc:1875] T 60eef9a26be04a579a4831488c958606 P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:35.298329 10589 ts_tablet_manager.cc:1875] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:35.300513 10588 ts_tablet_manager.cc:1888] T 60eef9a26be04a579a4831488c958606 P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:35.300521 10590 ts_tablet_manager.cc:1888] T 60eef9a26be04a579a4831488c958606 P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:35.300585 10588 log.cc:1196] T 60eef9a26be04a579a4831488c958606 P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/60eef9a26be04a579a4831488c958606
I20240503 13:59:35.300593 10590 log.cc:1196] T 60eef9a26be04a579a4831488c958606 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/60eef9a26be04a579a4831488c958606
I20240503 13:59:35.300637 10589 ts_tablet_manager.cc:1888] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:35.300776 10589 log.cc:1196] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/60eef9a26be04a579a4831488c958606
I20240503 13:59:35.300894 10588 ts_tablet_manager.cc:1909] T 60eef9a26be04a579a4831488c958606 P ab1d21935d6049888eb0ceccc3c9d735: Deleting consensus metadata
I20240503 13:59:35.301383 10590 ts_tablet_manager.cc:1909] T 60eef9a26be04a579a4831488c958606 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting consensus metadata
I20240503 13:59:35.301400 10589 ts_tablet_manager.cc:1909] T 60eef9a26be04a579a4831488c958606 P 0d77752181e944698e809e325e5937e6: Deleting consensus metadata
I20240503 13:59:35.302037 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 60eef9a26be04a579a4831488c958606 (table rebalancer_dynamic_table_new_name [id=daa36562f64a42fc8950d4fa0cb21dfe]) successfully deleted
I20240503 13:59:35.302054 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet 60eef9a26be04a579a4831488c958606 (table rebalancer_dynamic_table_new_name [id=daa36562f64a42fc8950d4fa0cb21dfe]) successfully deleted
I20240503 13:59:35.307502 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet 60eef9a26be04a579a4831488c958606 (table rebalancer_dynamic_table_new_name [id=daa36562f64a42fc8950d4fa0cb21dfe]) successfully deleted
I20240503 13:59:35.346515 10534 consensus_queue.cc:1035] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:35.349805 10552 consensus_queue.cc:1035] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:35.397845 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:35.402328 10552 consensus_queue.cc:1035] T bea48553d8c546abbb58f4d488e6b09a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:35.403604 8926 tablet_service.cc:1467] Processing CreateTablet for tablet 76f974b5fe534b81add519fa160e234d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=58ba6140fdcc4dffb1180b8861b4f469]), partition=
I20240503 13:59:35.403863 8926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f974b5fe534b81add519fa160e234d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.405822 10595 tablet_bootstrap.cc:492] T 76f974b5fe534b81add519fa160e234d P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:35.405864 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 76f974b5fe534b81add519fa160e234d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=58ba6140fdcc4dffb1180b8861b4f469]), partition=
I20240503 13:59:35.406098 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f974b5fe534b81add519fa160e234d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.406762 10595 tablet_bootstrap.cc:654] T 76f974b5fe534b81add519fa160e234d P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.407567 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 76f974b5fe534b81add519fa160e234d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=58ba6140fdcc4dffb1180b8861b4f469]), partition=
I20240503 13:59:35.407792 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f974b5fe534b81add519fa160e234d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.408387 10597 tablet_bootstrap.cc:492] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:35.409083 10597 tablet_bootstrap.cc:654] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.410108 10597 tablet_bootstrap.cc:492] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:35.410300 10597 ts_tablet_manager.cc:1367] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:35.410316 10599 tablet_bootstrap.cc:492] T 76f974b5fe534b81add519fa160e234d P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:35.410652 10597 raft_consensus.cc:348] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:35.410737 10597 raft_consensus.cc:374] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.410758 10597 raft_consensus.cc:729] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.410876 10597 consensus_queue.cc:260] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:35.411111 10597 ts_tablet_manager.cc:1398] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:35.411540 10599 tablet_bootstrap.cc:654] T 76f974b5fe534b81add519fa160e234d P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.413208 10535 consensus_queue.cc:1035] T bea48553d8c546abbb58f4d488e6b09a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:35.415735 10595 tablet_bootstrap.cc:492] T 76f974b5fe534b81add519fa160e234d P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:35.415856 10595 ts_tablet_manager.cc:1367] T 76f974b5fe534b81add519fa160e234d P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 13:59:35.416462 10595 raft_consensus.cc:348] T 76f974b5fe534b81add519fa160e234d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:35.416595 10595 raft_consensus.cc:374] T 76f974b5fe534b81add519fa160e234d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.416671 10595 raft_consensus.cc:729] T 76f974b5fe534b81add519fa160e234d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.416811 10595 consensus_queue.cc:260] T 76f974b5fe534b81add519fa160e234d P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:35.417131 10595 ts_tablet_manager.cc:1398] T 76f974b5fe534b81add519fa160e234d P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:35.420199 10599 tablet_bootstrap.cc:492] T 76f974b5fe534b81add519fa160e234d P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:35.420287 10599 ts_tablet_manager.cc:1367] T 76f974b5fe534b81add519fa160e234d P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 13:59:35.420641 10599 raft_consensus.cc:348] T 76f974b5fe534b81add519fa160e234d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:35.420728 10599 raft_consensus.cc:374] T 76f974b5fe534b81add519fa160e234d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.420764 10599 raft_consensus.cc:729] T 76f974b5fe534b81add519fa160e234d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.420908 10599 consensus_queue.cc:260] T 76f974b5fe534b81add519fa160e234d P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:35.421530 10599 ts_tablet_manager.cc:1398] T 76f974b5fe534b81add519fa160e234d P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:35.439592 10514 raft_consensus.cc:482] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:35.439713 10514 raft_consensus.cc:504] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:35.440109 10514 leader_election.cc:288] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:35.440464 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f974b5fe534b81add519fa160e234d" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:35.440515 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f974b5fe534b81add519fa160e234d" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:35.440589 9081 raft_consensus.cc:2454] T 76f974b5fe534b81add519fa160e234d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:35.440666 8947 raft_consensus.cc:2454] T 76f974b5fe534b81add519fa160e234d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:35.440876 6595 leader_election.cc:302] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:35.441147 10514 raft_consensus.cc:2781] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:35.441198 10514 raft_consensus.cc:482] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:35.441252 10514 raft_consensus.cc:3037] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.442273 10514 raft_consensus.cc:504] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:35.442602 10514 leader_election.cc:288] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:35.442968 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f974b5fe534b81add519fa160e234d" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:35.443086 9081 raft_consensus.cc:3037] T 76f974b5fe534b81add519fa160e234d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.443387 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f974b5fe534b81add519fa160e234d" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:35.443502 8947 raft_consensus.cc:3037] T 76f974b5fe534b81add519fa160e234d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.444114 9081 raft_consensus.cc:2454] T 76f974b5fe534b81add519fa160e234d P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:35.444393 6595 leader_election.cc:302] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:35.444512 8947 raft_consensus.cc:2454] T 76f974b5fe534b81add519fa160e234d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:35.444545 10514 raft_consensus.cc:2781] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:35.444612 10514 raft_consensus.cc:686] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 13:59:35.444751 10514 consensus_queue.cc:237] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:35.446632 6380 catalog_manager.cc:5484] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:35.555512 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:35.559753 9081 raft_consensus.cc:1261] T 76f974b5fe534b81add519fa160e234d P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:35.559792 8947 raft_consensus.cc:1261] T 76f974b5fe534b81add519fa160e234d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:35.560333 10605 consensus_queue.cc:1035] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:35.560639 10514 consensus_queue.cc:1035] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:35.563038 6380 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet 76f974b5fe534b81add519fa160e234d (table rebalancer_dynamic_table [id=58ba6140fdcc4dffb1180b8861b4f469]). Expected version 1 got 0
I20240503 13:59:35.563391 6383 catalog_manager.cc:5576] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481) does not have the latest schema for tablet 76f974b5fe534b81add519fa160e234d (table rebalancer_dynamic_table [id=58ba6140fdcc4dffb1180b8861b4f469]). Expected version 1 got 0
I20240503 13:59:35.562908 10608 tablet.cc:1722] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:35.563985 10608 tablet.cc:1620] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43: MemRowSet was empty: no flush needed.
I20240503 13:59:35.564643 6383 catalog_manager.cc:5576] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119) does not have the latest schema for tablet 76f974b5fe534b81add519fa160e234d (table rebalancer_dynamic_table [id=58ba6140fdcc4dffb1180b8861b4f469]). Expected version 1 got 0
I20240503 13:59:35.564662 10544 tablet.cc:1722] T 76f974b5fe534b81add519fa160e234d P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:35.564970 10544 tablet.cc:1620] T 76f974b5fe534b81add519fa160e234d P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:35.567164 6383 catalog_manager.cc:5576] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481) does not have the latest schema for tablet 76f974b5fe534b81add519fa160e234d (table rebalancer_dynamic_table [id=58ba6140fdcc4dffb1180b8861b4f469]). Expected version 1 got 0
I20240503 13:59:35.569638 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=58ba6140fdcc4dffb1180b8861b4f469] alter complete (version 1)
I20240503 13:59:35.570284 10609 tablet.cc:1722] T 76f974b5fe534b81add519fa160e234d P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:35.571487 10609 tablet.cc:1620] T 76f974b5fe534b81add519fa160e234d P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:35.579496 10564 consensus_queue.cc:1035] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:35.582082 10564 consensus_queue.cc:1035] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:35.603121 6380 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:35.608558 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 9809e66670ff47f4bb0845304dcea025 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=9ae2d84be5cd4571b93dd93183bc33dd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.608728 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 9809e66670ff47f4bb0845304dcea025 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=9ae2d84be5cd4571b93dd93183bc33dd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.608829 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9809e66670ff47f4bb0845304dcea025. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.609032 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9809e66670ff47f4bb0845304dcea025. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.609697 9322 tablet_service.cc:1467] Processing CreateTablet for tablet 9809e66670ff47f4bb0845304dcea025 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=9ae2d84be5cd4571b93dd93183bc33dd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.609768 9323 tablet_service.cc:1467] Processing CreateTablet for tablet a1faecbedbeb4cd685786c1bbd7f5be8 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=9ae2d84be5cd4571b93dd93183bc33dd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.609962 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9809e66670ff47f4bb0845304dcea025. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.610128 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1faecbedbeb4cd685786c1bbd7f5be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.610390 9321 tablet_service.cc:1467] Processing CreateTablet for tablet 0fe53d65ab4d482aad118d6e3bd5533c (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=9ae2d84be5cd4571b93dd93183bc33dd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.610620 9321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fe53d65ab4d482aad118d6e3bd5533c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.611193 6640 tablet_service.cc:1467] Processing CreateTablet for tablet a1faecbedbeb4cd685786c1bbd7f5be8 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=9ae2d84be5cd4571b93dd93183bc33dd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.611506 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1faecbedbeb4cd685786c1bbd7f5be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.612273 10613 tablet_bootstrap.cc:492] T a1faecbedbeb4cd685786c1bbd7f5be8 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:35.613287 10613 tablet_bootstrap.cc:654] T a1faecbedbeb4cd685786c1bbd7f5be8 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.613564 10599 tablet_bootstrap.cc:492] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:35.614420 10613 tablet_bootstrap.cc:492] T a1faecbedbeb4cd685786c1bbd7f5be8 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:35.614482 10599 tablet_bootstrap.cc:654] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.614491 10613 ts_tablet_manager.cc:1367] T a1faecbedbeb4cd685786c1bbd7f5be8 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:35.614722 10612 tablet_bootstrap.cc:492] T 9809e66670ff47f4bb0845304dcea025 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:35.614921 10613 raft_consensus.cc:348] T a1faecbedbeb4cd685786c1bbd7f5be8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:35.615039 10613 raft_consensus.cc:374] T a1faecbedbeb4cd685786c1bbd7f5be8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.615070 10613 raft_consensus.cc:729] T a1faecbedbeb4cd685786c1bbd7f5be8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.615088 10597 tablet_bootstrap.cc:492] T a1faecbedbeb4cd685786c1bbd7f5be8 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:35.615219 10613 consensus_queue.cc:260] T a1faecbedbeb4cd685786c1bbd7f5be8 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:35.615517 10613 ts_tablet_manager.cc:1398] T a1faecbedbeb4cd685786c1bbd7f5be8 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:35.615658 10613 tablet_bootstrap.cc:492] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:35.615896 10597 tablet_bootstrap.cc:654] T a1faecbedbeb4cd685786c1bbd7f5be8 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.616575 10612 tablet_bootstrap.cc:654] T 9809e66670ff47f4bb0845304dcea025 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.617712 10612 tablet_bootstrap.cc:492] T 9809e66670ff47f4bb0845304dcea025 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:35.617812 10612 ts_tablet_manager.cc:1367] T 9809e66670ff47f4bb0845304dcea025 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:35.617851 10613 tablet_bootstrap.cc:654] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.618286 10612 raft_consensus.cc:348] T 9809e66670ff47f4bb0845304dcea025 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:35.618387 10597 tablet_bootstrap.cc:492] T a1faecbedbeb4cd685786c1bbd7f5be8 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:35.618414 10612 raft_consensus.cc:374] T 9809e66670ff47f4bb0845304dcea025 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.618453 10612 raft_consensus.cc:729] T 9809e66670ff47f4bb0845304dcea025 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.618466 10597 ts_tablet_manager.cc:1367] T a1faecbedbeb4cd685786c1bbd7f5be8 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:35.618580 10612 consensus_queue.cc:260] T 9809e66670ff47f4bb0845304dcea025 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:35.618876 10612 ts_tablet_manager.cc:1398] T 9809e66670ff47f4bb0845304dcea025 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:35.618855 10597 raft_consensus.cc:348] T a1faecbedbeb4cd685786c1bbd7f5be8 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:35.618932 10597 raft_consensus.cc:374] T a1faecbedbeb4cd685786c1bbd7f5be8 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.618958 10597 raft_consensus.cc:729] T a1faecbedbeb4cd685786c1bbd7f5be8 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.619036 10613 tablet_bootstrap.cc:492] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:35.619098 10613 ts_tablet_manager.cc:1367] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:35.619060 10597 consensus_queue.cc:260] T a1faecbedbeb4cd685786c1bbd7f5be8 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:35.619352 10597 ts_tablet_manager.cc:1398] T a1faecbedbeb4cd685786c1bbd7f5be8 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:35.619673 10613 raft_consensus.cc:348] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:35.619908 10613 raft_consensus.cc:374] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.620111 10613 raft_consensus.cc:729] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.620321 10613 consensus_queue.cc:260] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:35.620785 10613 ts_tablet_manager.cc:1398] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:35.620918 9190 tablet_service.cc:1467] Processing CreateTablet for tablet a1faecbedbeb4cd685786c1bbd7f5be8 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=9ae2d84be5cd4571b93dd93183bc33dd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.620985 10613 tablet_bootstrap.cc:492] T 9809e66670ff47f4bb0845304dcea025 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:35.620946 9191 tablet_service.cc:1467] Processing CreateTablet for tablet 0fe53d65ab4d482aad118d6e3bd5533c (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=9ae2d84be5cd4571b93dd93183bc33dd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.621186 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1faecbedbeb4cd685786c1bbd7f5be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.621928 10613 tablet_bootstrap.cc:654] T 9809e66670ff47f4bb0845304dcea025 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.622937 10599 tablet_bootstrap.cc:492] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:35.623018 10599 ts_tablet_manager.cc:1367] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 13:59:35.623562 9191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fe53d65ab4d482aad118d6e3bd5533c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.623494 10599 raft_consensus.cc:348] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:35.623611 10599 raft_consensus.cc:374] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.623642 10599 raft_consensus.cc:729] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.623764 10599 consensus_queue.cc:260] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:35.624035 10599 ts_tablet_manager.cc:1398] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:35.624585 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 0fe53d65ab4d482aad118d6e3bd5533c (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=9ae2d84be5cd4571b93dd93183bc33dd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.624773 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fe53d65ab4d482aad118d6e3bd5533c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.625697 10620 tablet_bootstrap.cc:492] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:35.626504 10620 tablet_bootstrap.cc:654] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.626585 10613 tablet_bootstrap.cc:492] T 9809e66670ff47f4bb0845304dcea025 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:35.626722 10613 ts_tablet_manager.cc:1367] T 9809e66670ff47f4bb0845304dcea025 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:59:35.626924 10599 tablet_bootstrap.cc:492] T 0fe53d65ab4d482aad118d6e3bd5533c P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:35.627401 10613 raft_consensus.cc:348] T 9809e66670ff47f4bb0845304dcea025 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:35.627557 10613 raft_consensus.cc:374] T 9809e66670ff47f4bb0845304dcea025 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.627573 10620 tablet_bootstrap.cc:492] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:35.627650 10620 ts_tablet_manager.cc:1367] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:35.627643 10613 raft_consensus.cc:729] T 9809e66670ff47f4bb0845304dcea025 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.627847 10599 tablet_bootstrap.cc:654] T 0fe53d65ab4d482aad118d6e3bd5533c P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.627791 10613 consensus_queue.cc:260] T 9809e66670ff47f4bb0845304dcea025 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:35.628106 10620 raft_consensus.cc:348] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:35.628196 10620 raft_consensus.cc:374] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.628219 10620 raft_consensus.cc:729] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.628321 10620 consensus_queue.cc:260] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:35.628566 10620 ts_tablet_manager.cc:1398] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:35.628713 10620 tablet_bootstrap.cc:492] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:35.628969 10613 ts_tablet_manager.cc:1398] T 9809e66670ff47f4bb0845304dcea025 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:35.629650 10620 tablet_bootstrap.cc:654] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.629896 10599 tablet_bootstrap.cc:492] T 0fe53d65ab4d482aad118d6e3bd5533c P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:35.629966 10599 ts_tablet_manager.cc:1367] T 0fe53d65ab4d482aad118d6e3bd5533c P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:35.630399 10599 raft_consensus.cc:348] T 0fe53d65ab4d482aad118d6e3bd5533c P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:35.630498 10599 raft_consensus.cc:374] T 0fe53d65ab4d482aad118d6e3bd5533c P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.630534 10599 raft_consensus.cc:729] T 0fe53d65ab4d482aad118d6e3bd5533c P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.630653 10599 consensus_queue.cc:260] T 0fe53d65ab4d482aad118d6e3bd5533c P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:35.630934 10599 ts_tablet_manager.cc:1398] T 0fe53d65ab4d482aad118d6e3bd5533c P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:35.633221 10620 tablet_bootstrap.cc:492] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:35.633378 10546 raft_consensus.cc:482] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:35.633416 10620 ts_tablet_manager.cc:1367] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 13:59:35.633463 10546 raft_consensus.cc:504] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:35.633765 10546 leader_election.cc:288] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:35.634135 10620 raft_consensus.cc:348] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:35.634353 10620 raft_consensus.cc:374] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.634315 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9809e66670ff47f4bb0845304dcea025" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:35.634385 10620 raft_consensus.cc:729] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.634444 6527 raft_consensus.cc:2454] T 9809e66670ff47f4bb0845304dcea025 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:35.634496 10620 consensus_queue.cc:260] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:35.634760 10620 ts_tablet_manager.cc:1398] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:35.635048 9016 leader_election.cc:302] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:35.635051 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9809e66670ff47f4bb0845304dcea025" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:35.635154 9343 raft_consensus.cc:2454] T 9809e66670ff47f4bb0845304dcea025 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:35.635432 10546 raft_consensus.cc:2781] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:35.635483 10546 raft_consensus.cc:482] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:35.635516 10546 raft_consensus.cc:3037] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.636430 10546 raft_consensus.cc:504] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:35.636751 10546 leader_election.cc:288] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:35.637006 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9809e66670ff47f4bb0845304dcea025" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:35.637101 6527 raft_consensus.cc:3037] T 9809e66670ff47f4bb0845304dcea025 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.638058 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9809e66670ff47f4bb0845304dcea025" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:35.638093 6527 raft_consensus.cc:2454] T 9809e66670ff47f4bb0845304dcea025 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:35.638163 9343 raft_consensus.cc:3037] T 9809e66670ff47f4bb0845304dcea025 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.638273 10564 raft_consensus.cc:482] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:35.638339 10564 raft_consensus.cc:504] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:35.638612 10564 leader_election.cc:288] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:35.638859 9016 leader_election.cc:302] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:35.639106 9343 raft_consensus.cc:2454] T 9809e66670ff47f4bb0845304dcea025 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:35.639636 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe53d65ab4d482aad118d6e3bd5533c" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:35.639750 9211 raft_consensus.cc:2454] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:35.639765 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe53d65ab4d482aad118d6e3bd5533c" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:35.639806 10546 raft_consensus.cc:2781] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:35.639865 9081 raft_consensus.cc:2454] T 0fe53d65ab4d482aad118d6e3bd5533c P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:35.639997 10546 raft_consensus.cc:686] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 13:59:35.640100 9278 leader_election.cc:302] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:35.640126 10546 consensus_queue.cc:237] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:35.640542 10564 raft_consensus.cc:2781] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:35.640594 10564 raft_consensus.cc:482] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:35.640631 10564 raft_consensus.cc:3037] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.641809 10564 raft_consensus.cc:504] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:35.642136 10564 leader_election.cc:288] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:35.642386 6383 catalog_manager.cc:5484] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:35.642458 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe53d65ab4d482aad118d6e3bd5533c" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:35.642557 9081 raft_consensus.cc:3037] T 0fe53d65ab4d482aad118d6e3bd5533c P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.643441 9081 raft_consensus.cc:2454] T 0fe53d65ab4d482aad118d6e3bd5533c P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:35.643685 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe53d65ab4d482aad118d6e3bd5533c" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:35.643718 9278 leader_election.cc:302] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:35.643783 9211 raft_consensus.cc:3037] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.644079 10564 raft_consensus.cc:2781] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:35.644156 10564 raft_consensus.cc:686] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:35.644259 10564 consensus_queue.cc:237] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:35.644713 9211 raft_consensus.cc:2454] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:35.646301 6379 catalog_manager.cc:5484] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:35.674386 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:35.678999 10608 tablet.cc:1722] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:35.679345 10608 tablet.cc:1620] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43: MemRowSet was empty: no flush needed.
I20240503 13:59:35.679487 10609 tablet.cc:1722] T 76f974b5fe534b81add519fa160e234d P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:35.679659 10544 tablet.cc:1722] T 76f974b5fe534b81add519fa160e234d P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:35.679782 10609 tablet.cc:1620] T 76f974b5fe534b81add519fa160e234d P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:35.679967 10544 tablet.cc:1620] T 76f974b5fe534b81add519fa160e234d P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:35.685130 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=58ba6140fdcc4dffb1180b8861b4f469] alter complete (version 2)
I20240503 13:59:35.688028 10542 raft_consensus.cc:482] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:35.688134 10542 raft_consensus.cc:504] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:35.688450 10542 leader_election.cc:288] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:35.688735 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1faecbedbeb4cd685786c1bbd7f5be8" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:35.688787 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1faecbedbeb4cd685786c1bbd7f5be8" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:35.688858 6659 raft_consensus.cc:2454] T a1faecbedbeb4cd685786c1bbd7f5be8 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:35.688918 9343 raft_consensus.cc:2454] T a1faecbedbeb4cd685786c1bbd7f5be8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:35.689155 9148 leader_election.cc:302] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:35.689414 10542 raft_consensus.cc:2781] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:35.689478 10542 raft_consensus.cc:482] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:35.689524 10542 raft_consensus.cc:3037] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.690353 10542 raft_consensus.cc:504] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:35.690671 10542 leader_election.cc:288] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Requested vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:35.690958 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1faecbedbeb4cd685786c1bbd7f5be8" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:35.691054 9343 raft_consensus.cc:3037] T a1faecbedbeb4cd685786c1bbd7f5be8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.691067 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1faecbedbeb4cd685786c1bbd7f5be8" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:35.691161 6659 raft_consensus.cc:3037] T a1faecbedbeb4cd685786c1bbd7f5be8 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.691843 9343 raft_consensus.cc:2454] T a1faecbedbeb4cd685786c1bbd7f5be8 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:35.691996 6659 raft_consensus.cc:2454] T a1faecbedbeb4cd685786c1bbd7f5be8 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:35.692102 9148 leader_election.cc:302] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:35.692284 10542 raft_consensus.cc:2781] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:35.692359 10542 raft_consensus.cc:686] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 13:59:35.692502 10542 consensus_queue.cc:237] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:35.694237 6386 catalog_manager.cc:5484] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:35.786762 6386 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:35.791755 10608 tablet.cc:1722] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:35.792143 10544 tablet.cc:1722] T 76f974b5fe534b81add519fa160e234d P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:35.792922 10609 tablet.cc:1722] T 76f974b5fe534b81add519fa160e234d P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:35.798007 6383 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=58ba6140fdcc4dffb1180b8861b4f469] alter complete (version 3)
I20240503 13:59:35.900172 6379 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:35.900349 6379 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:35.902312 6379 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 76f974b5fe534b81add519fa160e234d
I20240503 13:59:35.902941 9058 tablet_service.cc:1514] Processing DeleteTablet for tablet 76f974b5fe534b81add519fa160e234d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:35 UTC) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:35.902972 8926 tablet_service.cc:1514] Processing DeleteTablet for tablet 76f974b5fe534b81add519fa160e234d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:35 UTC) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:35.903117 6640 tablet_service.cc:1514] Processing DeleteTablet for tablet 76f974b5fe534b81add519fa160e234d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:35 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:35.903344 10629 tablet_replica.cc:330] T 76f974b5fe534b81add519fa160e234d P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:35.903375 10630 tablet_replica.cc:330] T 76f974b5fe534b81add519fa160e234d P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:35.903512 10630 raft_consensus.cc:2229] T 76f974b5fe534b81add519fa160e234d P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:35.903519 10629 raft_consensus.cc:2229] T 76f974b5fe534b81add519fa160e234d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:35.903529 10631 tablet_replica.cc:330] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:35.903648 10630 raft_consensus.cc:2258] T 76f974b5fe534b81add519fa160e234d P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:35.903676 10631 raft_consensus.cc:2229] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:35.903705 10629 raft_consensus.cc:2258] T 76f974b5fe534b81add519fa160e234d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:35.904296 10631 raft_consensus.cc:2258] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:35.904304 10630 ts_tablet_manager.cc:1875] T 76f974b5fe534b81add519fa160e234d P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:35.904546 10629 ts_tablet_manager.cc:1875] T 76f974b5fe534b81add519fa160e234d P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:35.904985 10631 ts_tablet_manager.cc:1875] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:35.906667 10630 ts_tablet_manager.cc:1888] T 76f974b5fe534b81add519fa160e234d P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:35.906829 10630 log.cc:1196] T 76f974b5fe534b81add519fa160e234d P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/76f974b5fe534b81add519fa160e234d
I20240503 13:59:35.906847 10629 ts_tablet_manager.cc:1888] T 76f974b5fe534b81add519fa160e234d P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:35.906929 10629 log.cc:1196] T 76f974b5fe534b81add519fa160e234d P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/76f974b5fe534b81add519fa160e234d
I20240503 13:59:35.907131 10630 ts_tablet_manager.cc:1909] T 76f974b5fe534b81add519fa160e234d P 8d574fb4a89d4eea947c859470d44dd4: Deleting consensus metadata
I20240503 13:59:35.907303 10629 ts_tablet_manager.cc:1909] T 76f974b5fe534b81add519fa160e234d P 06ca8dbc004c42b99faa527a4e6f9545: Deleting consensus metadata
I20240503 13:59:35.908236 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet 76f974b5fe534b81add519fa160e234d (table rebalancer_dynamic_table_new_name [id=58ba6140fdcc4dffb1180b8861b4f469]) successfully deleted
I20240503 13:59:35.908298 10631 ts_tablet_manager.cc:1888] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:35.908354 10631 log.cc:1196] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/76f974b5fe534b81add519fa160e234d
I20240503 13:59:35.908418 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 76f974b5fe534b81add519fa160e234d (table rebalancer_dynamic_table_new_name [id=58ba6140fdcc4dffb1180b8861b4f469]) successfully deleted
I20240503 13:59:35.908694 10631 ts_tablet_manager.cc:1909] T 76f974b5fe534b81add519fa160e234d P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:35.909360 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet 76f974b5fe534b81add519fa160e234d (table rebalancer_dynamic_table_new_name [id=58ba6140fdcc4dffb1180b8861b4f469]) successfully deleted
I20240503 13:59:36.004403 6379 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:36.009811 9322 tablet_service.cc:1467] Processing CreateTablet for tablet ea06d120aef147b385051bca1b131f93 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b0147efe5605494a8e2170ae7eb28ef0]), partition=
I20240503 13:59:36.010088 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea06d120aef147b385051bca1b131f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.010272 9058 tablet_service.cc:1467] Processing CreateTablet for tablet ea06d120aef147b385051bca1b131f93 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b0147efe5605494a8e2170ae7eb28ef0]), partition=
I20240503 13:59:36.010502 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea06d120aef147b385051bca1b131f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.012377 10599 tablet_bootstrap.cc:492] T ea06d120aef147b385051bca1b131f93 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:36.012872 9191 tablet_service.cc:1467] Processing CreateTablet for tablet ea06d120aef147b385051bca1b131f93 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b0147efe5605494a8e2170ae7eb28ef0]), partition=
I20240503 13:59:36.013073 9191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea06d120aef147b385051bca1b131f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.013352 10599 tablet_bootstrap.cc:654] T ea06d120aef147b385051bca1b131f93 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.014549 10599 tablet_bootstrap.cc:492] T ea06d120aef147b385051bca1b131f93 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:36.014634 10599 ts_tablet_manager.cc:1367] T ea06d120aef147b385051bca1b131f93 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:36.014722 10613 tablet_bootstrap.cc:492] T ea06d120aef147b385051bca1b131f93 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:36.014997 10599 raft_consensus.cc:348] T ea06d120aef147b385051bca1b131f93 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:36.015142 10599 raft_consensus.cc:374] T ea06d120aef147b385051bca1b131f93 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.015177 10599 raft_consensus.cc:729] T ea06d120aef147b385051bca1b131f93 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.015298 10599 consensus_queue.cc:260] T ea06d120aef147b385051bca1b131f93 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:36.015513 10613 tablet_bootstrap.cc:654] T ea06d120aef147b385051bca1b131f93 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.015578 10599 ts_tablet_manager.cc:1398] T ea06d120aef147b385051bca1b131f93 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:36.015751 10620 tablet_bootstrap.cc:492] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:36.016602 10620 tablet_bootstrap.cc:654] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.017030 10613 tablet_bootstrap.cc:492] T ea06d120aef147b385051bca1b131f93 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:36.017109 10613 ts_tablet_manager.cc:1367] T ea06d120aef147b385051bca1b131f93 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:36.017652 10620 tablet_bootstrap.cc:492] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:36.017593 10613 raft_consensus.cc:348] T ea06d120aef147b385051bca1b131f93 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:36.017699 10613 raft_consensus.cc:374] T ea06d120aef147b385051bca1b131f93 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.017737 10613 raft_consensus.cc:729] T ea06d120aef147b385051bca1b131f93 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.017746 10620 ts_tablet_manager.cc:1367] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:36.017874 10613 consensus_queue.cc:260] T ea06d120aef147b385051bca1b131f93 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:36.018172 10613 ts_tablet_manager.cc:1398] T ea06d120aef147b385051bca1b131f93 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:36.018185 10620 raft_consensus.cc:348] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:36.018302 10620 raft_consensus.cc:374] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.018347 10620 raft_consensus.cc:729] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.018460 10620 consensus_queue.cc:260] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:36.018720 10620 ts_tablet_manager.cc:1398] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:36.018851 10542 raft_consensus.cc:482] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:36.018910 10542 raft_consensus.cc:504] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:36.019336 10542 leader_election.cc:288] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:36.021214 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea06d120aef147b385051bca1b131f93" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:36.021281 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea06d120aef147b385051bca1b131f93" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:36.021335 9343 raft_consensus.cc:2454] T ea06d120aef147b385051bca1b131f93 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:36.021394 9081 raft_consensus.cc:2454] T ea06d120aef147b385051bca1b131f93 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:36.021608 9148 leader_election.cc:302] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:36.021817 10542 raft_consensus.cc:2781] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:36.021863 10542 raft_consensus.cc:482] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:36.021899 10542 raft_consensus.cc:3037] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.022815 10542 raft_consensus.cc:504] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:36.023240 10542 leader_election.cc:288] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:36.023577 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea06d120aef147b385051bca1b131f93" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:36.023684 9343 raft_consensus.cc:3037] T ea06d120aef147b385051bca1b131f93 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.024103 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea06d120aef147b385051bca1b131f93" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:36.024209 9081 raft_consensus.cc:3037] T ea06d120aef147b385051bca1b131f93 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.024606 9343 raft_consensus.cc:2454] T ea06d120aef147b385051bca1b131f93 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:36.025076 9081 raft_consensus.cc:2454] T ea06d120aef147b385051bca1b131f93 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:36.025396 9148 leader_election.cc:302] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:36.025946 10304 raft_consensus.cc:2781] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:36.026036 10304 raft_consensus.cc:686] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 13:59:36.026154 10304 consensus_queue.cc:237] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:36.028926 6379 catalog_manager.cc:5484] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:36.056265 10624 consensus_queue.cc:1035] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:36.059340 10637 consensus_queue.cc:1035] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:36.136399 6383 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:36.137702 10339 consensus_queue.cc:1035] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:36.141360 10339 consensus_queue.cc:1035] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:36.144451 9081 raft_consensus.cc:1261] T ea06d120aef147b385051bca1b131f93 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:36.144804 10304 consensus_queue.cc:1035] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:36.145009 9342 raft_consensus.cc:1261] T ea06d120aef147b385051bca1b131f93 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:36.147428 10542 consensus_queue.cc:1035] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:36.149294 10304 consensus_queue.cc:1035] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:36.149410 10609 tablet.cc:1722] T ea06d120aef147b385051bca1b131f93 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:36.149619 6386 catalog_manager.cc:5576] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481) does not have the latest schema for tablet ea06d120aef147b385051bca1b131f93 (table rebalancer_dynamic_table [id=b0147efe5605494a8e2170ae7eb28ef0]). Expected version 1 got 0
I20240503 13:59:36.149706 10609 tablet.cc:1620] T ea06d120aef147b385051bca1b131f93 P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:36.154299 10304 consensus_queue.cc:1035] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:36.155077 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=b0147efe5605494a8e2170ae7eb28ef0] alter complete (version 1)
I20240503 13:59:36.156162 6386 catalog_manager.cc:5576] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091) does not have the latest schema for tablet ea06d120aef147b385051bca1b131f93 (table rebalancer_dynamic_table [id=b0147efe5605494a8e2170ae7eb28ef0]). Expected version 1 got 0
I20240503 13:59:36.156564 10646 tablet.cc:1722] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:36.157325 10646 tablet.cc:1620] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:36.157550 6383 catalog_manager.cc:5576] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931) does not have the latest schema for tablet ea06d120aef147b385051bca1b131f93 (table rebalancer_dynamic_table [id=b0147efe5605494a8e2170ae7eb28ef0]). Expected version 1 got 0
I20240503 13:59:36.158766 10650 tablet.cc:1722] T ea06d120aef147b385051bca1b131f93 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:36.159073 10650 tablet.cc:1620] T ea06d120aef147b385051bca1b131f93 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:36.159615 6383 catalog_manager.cc:5576] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931) does not have the latest schema for tablet ea06d120aef147b385051bca1b131f93 (table rebalancer_dynamic_table [id=b0147efe5605494a8e2170ae7eb28ef0]). Expected version 1 got 0
I20240503 13:59:36.206107 6386 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:36.213021 9191 tablet_service.cc:1467] Processing CreateTablet for tablet 44eddd53c3f348a9a1117a20473cbb65 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=2c584aa1fb5a4bc4a40f13eaf00c9ddf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:36.213325 9191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44eddd53c3f348a9a1117a20473cbb65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.213374 9190 tablet_service.cc:1467] Processing CreateTablet for tablet b7e1060596204c60882388e984986ac6 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=2c584aa1fb5a4bc4a40f13eaf00c9ddf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:36.213583 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7e1060596204c60882388e984986ac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.214277 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 44eddd53c3f348a9a1117a20473cbb65 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=2c584aa1fb5a4bc4a40f13eaf00c9ddf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:36.214562 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44eddd53c3f348a9a1117a20473cbb65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.215370 10620 tablet_bootstrap.cc:492] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:36.215768 6769 tablet_service.cc:1467] Processing CreateTablet for tablet b7e1060596204c60882388e984986ac6 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=2c584aa1fb5a4bc4a40f13eaf00c9ddf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:36.215868 9322 tablet_service.cc:1467] Processing CreateTablet for tablet 44eddd53c3f348a9a1117a20473cbb65 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=2c584aa1fb5a4bc4a40f13eaf00c9ddf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:36.216025 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7e1060596204c60882388e984986ac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.216084 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44eddd53c3f348a9a1117a20473cbb65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.216318 10620 tablet_bootstrap.cc:654] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.216809 10599 tablet_bootstrap.cc:492] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:36.217779 10599 tablet_bootstrap.cc:654] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.218290 8926 tablet_service.cc:1467] Processing CreateTablet for tablet 9358fd0a35c94daabbad58fe99547320 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=2c584aa1fb5a4bc4a40f13eaf00c9ddf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:36.218578 8926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9358fd0a35c94daabbad58fe99547320. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.218669 6902 tablet_service.cc:1467] Processing CreateTablet for tablet 9358fd0a35c94daabbad58fe99547320 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=2c584aa1fb5a4bc4a40f13eaf00c9ddf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:36.218957 6902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9358fd0a35c94daabbad58fe99547320. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.220430 10613 tablet_bootstrap.cc:492] T 44eddd53c3f348a9a1117a20473cbb65 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:36.220615 10654 tablet_bootstrap.cc:492] T 9358fd0a35c94daabbad58fe99547320 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:36.221125 10655 tablet_bootstrap.cc:492] T 9358fd0a35c94daabbad58fe99547320 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:36.221343 10613 tablet_bootstrap.cc:654] T 44eddd53c3f348a9a1117a20473cbb65 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.221462 10599 tablet_bootstrap.cc:492] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:36.221537 10599 ts_tablet_manager.cc:1367] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:36.221738 10654 tablet_bootstrap.cc:654] T 9358fd0a35c94daabbad58fe99547320 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.221982 10599 raft_consensus.cc:348] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:36.222116 10655 tablet_bootstrap.cc:654] T 9358fd0a35c94daabbad58fe99547320 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.222119 10599 raft_consensus.cc:374] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.222170 10599 raft_consensus.cc:729] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.222317 10599 consensus_queue.cc:260] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:36.222620 10599 ts_tablet_manager.cc:1398] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:36.223579 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 9358fd0a35c94daabbad58fe99547320 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=2c584aa1fb5a4bc4a40f13eaf00c9ddf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:36.223778 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9358fd0a35c94daabbad58fe99547320. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.225119 10652 tablet_bootstrap.cc:492] T b7e1060596204c60882388e984986ac6 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:36.225332 10613 tablet_bootstrap.cc:492] T 44eddd53c3f348a9a1117a20473cbb65 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:36.225405 10613 ts_tablet_manager.cc:1367] T 44eddd53c3f348a9a1117a20473cbb65 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:36.225831 10613 raft_consensus.cc:348] T 44eddd53c3f348a9a1117a20473cbb65 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:36.225960 10613 raft_consensus.cc:374] T 44eddd53c3f348a9a1117a20473cbb65 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.225989 10652 tablet_bootstrap.cc:654] T b7e1060596204c60882388e984986ac6 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.225996 10613 raft_consensus.cc:729] T 44eddd53c3f348a9a1117a20473cbb65 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.226153 10599 tablet_bootstrap.cc:492] T 9358fd0a35c94daabbad58fe99547320 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:36.226122 10613 consensus_queue.cc:260] T 44eddd53c3f348a9a1117a20473cbb65 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:36.226454 10613 ts_tablet_manager.cc:1398] T 44eddd53c3f348a9a1117a20473cbb65 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:36.227035 10599 tablet_bootstrap.cc:654] T 9358fd0a35c94daabbad58fe99547320 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.227866 9322 tablet_service.cc:1467] Processing CreateTablet for tablet b7e1060596204c60882388e984986ac6 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=2c584aa1fb5a4bc4a40f13eaf00c9ddf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:36.228128 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7e1060596204c60882388e984986ac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.228412 10620 tablet_bootstrap.cc:492] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:36.228506 10620 ts_tablet_manager.cc:1367] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20240503 13:59:36.228977 10620 raft_consensus.cc:348] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:36.229075 10620 raft_consensus.cc:374] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.229106 10620 raft_consensus.cc:729] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.229131 10599 tablet_bootstrap.cc:492] T 9358fd0a35c94daabbad58fe99547320 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:36.229202 10599 ts_tablet_manager.cc:1367] T 9358fd0a35c94daabbad58fe99547320 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:36.229226 10620 consensus_queue.cc:260] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:36.229511 10620 ts_tablet_manager.cc:1398] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:36.229593 10599 raft_consensus.cc:348] T 9358fd0a35c94daabbad58fe99547320 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:36.229666 10620 tablet_bootstrap.cc:492] T 44eddd53c3f348a9a1117a20473cbb65 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:36.229676 10599 raft_consensus.cc:374] T 9358fd0a35c94daabbad58fe99547320 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.229707 10599 raft_consensus.cc:729] T 9358fd0a35c94daabbad58fe99547320 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.229826 10613 tablet_bootstrap.cc:492] T b7e1060596204c60882388e984986ac6 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:36.229802 10599 consensus_queue.cc:260] T 9358fd0a35c94daabbad58fe99547320 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:36.230609 10599 ts_tablet_manager.cc:1398] T 9358fd0a35c94daabbad58fe99547320 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:36.230886 10620 tablet_bootstrap.cc:654] T 44eddd53c3f348a9a1117a20473cbb65 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.230886 10613 tablet_bootstrap.cc:654] T b7e1060596204c60882388e984986ac6 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.232100 10620 tablet_bootstrap.cc:492] T 44eddd53c3f348a9a1117a20473cbb65 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:36.232161 10620 ts_tablet_manager.cc:1367] T 44eddd53c3f348a9a1117a20473cbb65 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:36.232524 10620 raft_consensus.cc:348] T 44eddd53c3f348a9a1117a20473cbb65 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:36.232609 10620 raft_consensus.cc:374] T 44eddd53c3f348a9a1117a20473cbb65 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.232635 10620 raft_consensus.cc:729] T 44eddd53c3f348a9a1117a20473cbb65 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.232743 10620 consensus_queue.cc:260] T 44eddd53c3f348a9a1117a20473cbb65 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:36.232879 10613 tablet_bootstrap.cc:492] T b7e1060596204c60882388e984986ac6 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:36.232949 10620 ts_tablet_manager.cc:1398] T 44eddd53c3f348a9a1117a20473cbb65 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:36.232961 10613 ts_tablet_manager.cc:1367] T b7e1060596204c60882388e984986ac6 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:36.233413 10613 raft_consensus.cc:348] T b7e1060596204c60882388e984986ac6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:36.233506 10613 raft_consensus.cc:374] T b7e1060596204c60882388e984986ac6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.233536 10613 raft_consensus.cc:729] T b7e1060596204c60882388e984986ac6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.233656 10613 consensus_queue.cc:260] T b7e1060596204c60882388e984986ac6 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:36.233932 10613 ts_tablet_manager.cc:1398] T b7e1060596204c60882388e984986ac6 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:36.236972 10655 tablet_bootstrap.cc:492] T 9358fd0a35c94daabbad58fe99547320 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:36.237107 10655 ts_tablet_manager.cc:1367] T 9358fd0a35c94daabbad58fe99547320 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 13:59:36.237150 10654 tablet_bootstrap.cc:492] T 9358fd0a35c94daabbad58fe99547320 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:36.237300 10654 ts_tablet_manager.cc:1367] T 9358fd0a35c94daabbad58fe99547320 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 13:59:36.237583 10655 raft_consensus.cc:348] T 9358fd0a35c94daabbad58fe99547320 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:36.237699 10655 raft_consensus.cc:374] T 9358fd0a35c94daabbad58fe99547320 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.237730 10655 raft_consensus.cc:729] T 9358fd0a35c94daabbad58fe99547320 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.237903 10654 raft_consensus.cc:348] T 9358fd0a35c94daabbad58fe99547320 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:36.238003 10654 raft_consensus.cc:374] T 9358fd0a35c94daabbad58fe99547320 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.238057 10654 raft_consensus.cc:729] T 9358fd0a35c94daabbad58fe99547320 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.238193 10654 consensus_queue.cc:260] T 9358fd0a35c94daabbad58fe99547320 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:36.239396 10655 consensus_queue.cc:260] T 9358fd0a35c94daabbad58fe99547320 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:36.240931 10654 ts_tablet_manager.cc:1398] T 9358fd0a35c94daabbad58fe99547320 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:36.241565 10655 ts_tablet_manager.cc:1398] T 9358fd0a35c94daabbad58fe99547320 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:36.248014 10652 tablet_bootstrap.cc:492] T b7e1060596204c60882388e984986ac6 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:36.248114 10652 ts_tablet_manager.cc:1367] T b7e1060596204c60882388e984986ac6 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.000s
I20240503 13:59:36.248581 10652 raft_consensus.cc:348] T b7e1060596204c60882388e984986ac6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:36.248747 10652 raft_consensus.cc:374] T b7e1060596204c60882388e984986ac6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.248790 10652 raft_consensus.cc:729] T b7e1060596204c60882388e984986ac6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.248935 10652 consensus_queue.cc:260] T b7e1060596204c60882388e984986ac6 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:36.249320 10652 ts_tablet_manager.cc:1398] T b7e1060596204c60882388e984986ac6 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:36.256963 6383 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:36.261689 10646 tablet.cc:1722] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:36.262042 10646 tablet.cc:1620] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:36.262107 10650 tablet.cc:1722] T ea06d120aef147b385051bca1b131f93 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:36.262406 10650 tablet.cc:1620] T ea06d120aef147b385051bca1b131f93 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:36.262979 10609 tablet.cc:1722] T ea06d120aef147b385051bca1b131f93 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:36.263250 10609 tablet.cc:1620] T ea06d120aef147b385051bca1b131f93 P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:36.267350 6383 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=b0147efe5605494a8e2170ae7eb28ef0] alter complete (version 2)
I20240503 13:59:36.273084 10637 raft_consensus.cc:482] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:36.273193 10637 raft_consensus.cc:504] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:36.273593 10637 leader_election.cc:288] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:36.273849 9342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44eddd53c3f348a9a1117a20473cbb65" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:36.273967 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44eddd53c3f348a9a1117a20473cbb65" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:36.273995 9342 raft_consensus.cc:2454] T 44eddd53c3f348a9a1117a20473cbb65 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:36.274089 9211 raft_consensus.cc:2454] T 44eddd53c3f348a9a1117a20473cbb65 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:36.274279 9017 leader_election.cc:302] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:36.274452 10637 raft_consensus.cc:2781] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:36.274514 10637 raft_consensus.cc:482] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:36.274552 10637 raft_consensus.cc:3037] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.275753 10637 raft_consensus.cc:504] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:36.276099 10637 leader_election.cc:288] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:36.276377 9342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44eddd53c3f348a9a1117a20473cbb65" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:36.276415 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44eddd53c3f348a9a1117a20473cbb65" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:36.276477 9342 raft_consensus.cc:3037] T 44eddd53c3f348a9a1117a20473cbb65 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.276513 9211 raft_consensus.cc:3037] T 44eddd53c3f348a9a1117a20473cbb65 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.277243 9211 raft_consensus.cc:2454] T 44eddd53c3f348a9a1117a20473cbb65 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:36.277290 9342 raft_consensus.cc:2454] T 44eddd53c3f348a9a1117a20473cbb65 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:36.277529 9016 leader_election.cc:302] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:36.277722 10637 raft_consensus.cc:2781] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:36.277797 10637 raft_consensus.cc:686] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 13:59:36.277935 10637 consensus_queue.cc:237] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:36.279946 6379 catalog_manager.cc:5484] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:36.311635 10637 raft_consensus.cc:482] T 9358fd0a35c94daabbad58fe99547320 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:36.311744 10637 raft_consensus.cc:504] T 9358fd0a35c94daabbad58fe99547320 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:36.312108 10637 leader_election.cc:288] T 9358fd0a35c94daabbad58fe99547320 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:36.312690 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9358fd0a35c94daabbad58fe99547320" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:36.312848 6922 raft_consensus.cc:2454] T 9358fd0a35c94daabbad58fe99547320 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:36.312925 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9358fd0a35c94daabbad58fe99547320" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:36.313056 8947 raft_consensus.cc:2454] T 9358fd0a35c94daabbad58fe99547320 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:36.313129 9016 leader_election.cc:302] T 9358fd0a35c94daabbad58fe99547320 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:36.313282 10637 raft_consensus.cc:2781] T 9358fd0a35c94daabbad58fe99547320 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:36.313351 10637 raft_consensus.cc:482] T 9358fd0a35c94daabbad58fe99547320 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:36.313405 10637 raft_consensus.cc:3037] T 9358fd0a35c94daabbad58fe99547320 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.315018 10637 raft_consensus.cc:504] T 9358fd0a35c94daabbad58fe99547320 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:36.315369 10637 leader_election.cc:288] T 9358fd0a35c94daabbad58fe99547320 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:36.315656 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9358fd0a35c94daabbad58fe99547320" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:36.315670 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9358fd0a35c94daabbad58fe99547320" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:36.315775 6922 raft_consensus.cc:3037] T 9358fd0a35c94daabbad58fe99547320 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.315771 8947 raft_consensus.cc:3037] T 9358fd0a35c94daabbad58fe99547320 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.316699 8947 raft_consensus.cc:2454] T 9358fd0a35c94daabbad58fe99547320 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:36.316803 6922 raft_consensus.cc:2454] T 9358fd0a35c94daabbad58fe99547320 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:36.317106 9016 leader_election.cc:302] T 9358fd0a35c94daabbad58fe99547320 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:36.317318 10637 raft_consensus.cc:2781] T 9358fd0a35c94daabbad58fe99547320 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:36.317391 10637 raft_consensus.cc:686] T 9358fd0a35c94daabbad58fe99547320 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 13:59:36.317504 10637 consensus_queue.cc:237] T 9358fd0a35c94daabbad58fe99547320 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:36.319499 6379 catalog_manager.cc:5484] T 9358fd0a35c94daabbad58fe99547320 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:36.368772 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:36.373510 10646 tablet.cc:1722] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:36.374007 10609 tablet.cc:1722] T ea06d120aef147b385051bca1b131f93 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:36.374131 10650 tablet.cc:1722] T ea06d120aef147b385051bca1b131f93 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:36.380494 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=b0147efe5605494a8e2170ae7eb28ef0] alter complete (version 3)
I20240503 13:59:36.484114 6379 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:36.484305 6379 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:36.486172 6379 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet ea06d120aef147b385051bca1b131f93
I20240503 13:59:36.486721 9058 tablet_service.cc:1514] Processing DeleteTablet for tablet ea06d120aef147b385051bca1b131f93 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:36 UTC) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:36.486809 9322 tablet_service.cc:1514] Processing DeleteTablet for tablet ea06d120aef147b385051bca1b131f93 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:36 UTC) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:36.487084 10666 tablet_replica.cc:330] T ea06d120aef147b385051bca1b131f93 P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:36.487238 10666 raft_consensus.cc:2229] T ea06d120aef147b385051bca1b131f93 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:36.487380 9191 tablet_service.cc:1514] Processing DeleteTablet for tablet ea06d120aef147b385051bca1b131f93 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:36 UTC) from {username='slave'} at 127.0.0.1:57960
I20240503 13:59:36.487452 10666 raft_consensus.cc:2258] T ea06d120aef147b385051bca1b131f93 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:36.487522 10667 tablet_replica.cc:330] T ea06d120aef147b385051bca1b131f93 P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:36.487663 10667 raft_consensus.cc:2229] T ea06d120aef147b385051bca1b131f93 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:36.487794 10667 raft_consensus.cc:2258] T ea06d120aef147b385051bca1b131f93 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:36.487820 10668 tablet_replica.cc:330] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 13:59:36.487953 10668 raft_consensus.cc:2229] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:36.488175 10668 raft_consensus.cc:2258] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:36.488459 10666 ts_tablet_manager.cc:1875] T ea06d120aef147b385051bca1b131f93 P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:36.488574 10667 ts_tablet_manager.cc:1875] T ea06d120aef147b385051bca1b131f93 P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:36.488782 10668 ts_tablet_manager.cc:1875] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:36.491091 10668 ts_tablet_manager.cc:1888] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:36.491158 10668 log.cc:1196] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/ea06d120aef147b385051bca1b131f93
I20240503 13:59:36.491179 10667 ts_tablet_manager.cc:1888] T ea06d120aef147b385051bca1b131f93 P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:36.491312 10667 log.cc:1196] T ea06d120aef147b385051bca1b131f93 P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/ea06d120aef147b385051bca1b131f93
I20240503 13:59:36.491494 10668 ts_tablet_manager.cc:1909] T ea06d120aef147b385051bca1b131f93 P ab1d21935d6049888eb0ceccc3c9d735: Deleting consensus metadata
I20240503 13:59:36.491515 10666 ts_tablet_manager.cc:1888] T ea06d120aef147b385051bca1b131f93 P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:36.491583 10666 log.cc:1196] T ea06d120aef147b385051bca1b131f93 P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/ea06d120aef147b385051bca1b131f93
I20240503 13:59:36.491634 10667 ts_tablet_manager.cc:1909] T ea06d120aef147b385051bca1b131f93 P 0d77752181e944698e809e325e5937e6: Deleting consensus metadata
I20240503 13:59:36.491849 10666 ts_tablet_manager.cc:1909] T ea06d120aef147b385051bca1b131f93 P 8d574fb4a89d4eea947c859470d44dd4: Deleting consensus metadata
I20240503 13:59:36.492143 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet ea06d120aef147b385051bca1b131f93 (table rebalancer_dynamic_table_new_name [id=b0147efe5605494a8e2170ae7eb28ef0]) successfully deleted
I20240503 13:59:36.492339 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet ea06d120aef147b385051bca1b131f93 (table rebalancer_dynamic_table_new_name [id=b0147efe5605494a8e2170ae7eb28ef0]) successfully deleted
I20240503 13:59:36.492389 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet ea06d120aef147b385051bca1b131f93 (table rebalancer_dynamic_table_new_name [id=b0147efe5605494a8e2170ae7eb28ef0]) successfully deleted
I20240503 13:59:36.496480 10542 raft_consensus.cc:482] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:36.496570 10542 raft_consensus.cc:504] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:36.497056 10542 leader_election.cc:288] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:36.497345 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e1060596204c60882388e984986ac6" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:36.497385 9342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e1060596204c60882388e984986ac6" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:36.497460 6790 raft_consensus.cc:2454] T b7e1060596204c60882388e984986ac6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:36.497519 9342 raft_consensus.cc:2454] T b7e1060596204c60882388e984986ac6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:36.497813 9146 leader_election.cc:302] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:36.497949 10542 raft_consensus.cc:2781] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:36.498003 10542 raft_consensus.cc:482] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:36.498029 10542 raft_consensus.cc:3037] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.498816 10542 raft_consensus.cc:504] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:36.499130 10542 leader_election.cc:288] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:36.499601 9342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e1060596204c60882388e984986ac6" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:36.499732 9342 raft_consensus.cc:3037] T b7e1060596204c60882388e984986ac6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.499682 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e1060596204c60882388e984986ac6" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:36.499830 6790 raft_consensus.cc:3037] T b7e1060596204c60882388e984986ac6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.500730 6790 raft_consensus.cc:2454] T b7e1060596204c60882388e984986ac6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:36.500730 9342 raft_consensus.cc:2454] T b7e1060596204c60882388e984986ac6 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:36.501147 9148 leader_election.cc:302] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:36.501327 10542 raft_consensus.cc:2781] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:36.501415 10542 raft_consensus.cc:686] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 13:59:36.501546 10542 consensus_queue.cc:237] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:36.503641 6379 catalog_manager.cc:5484] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:36.588694 6379 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:36.593750 9191 tablet_service.cc:1467] Processing CreateTablet for tablet 48f4038e5090490580fa9acf38cb7404 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3d54da85efa540799bd125c75f2b7a7d]), partition=
I20240503 13:59:36.594010 9191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f4038e5090490580fa9acf38cb7404. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.595667 6769 tablet_service.cc:1467] Processing CreateTablet for tablet 48f4038e5090490580fa9acf38cb7404 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3d54da85efa540799bd125c75f2b7a7d]), partition=
I20240503 13:59:36.595891 8926 tablet_service.cc:1467] Processing CreateTablet for tablet 48f4038e5090490580fa9acf38cb7404 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3d54da85efa540799bd125c75f2b7a7d]), partition=
I20240503 13:59:36.595918 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f4038e5090490580fa9acf38cb7404. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.596107 8926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f4038e5090490580fa9acf38cb7404. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.597712 10652 tablet_bootstrap.cc:492] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:36.597780 10620 tablet_bootstrap.cc:492] T 48f4038e5090490580fa9acf38cb7404 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:36.598726 10654 tablet_bootstrap.cc:492] T 48f4038e5090490580fa9acf38cb7404 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:36.598754 10620 tablet_bootstrap.cc:654] T 48f4038e5090490580fa9acf38cb7404 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.599460 10654 tablet_bootstrap.cc:654] T 48f4038e5090490580fa9acf38cb7404 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.599998 10620 tablet_bootstrap.cc:492] T 48f4038e5090490580fa9acf38cb7404 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:36.600088 10620 ts_tablet_manager.cc:1367] T 48f4038e5090490580fa9acf38cb7404 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:36.600229 10652 tablet_bootstrap.cc:654] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.600976 10620 raft_consensus.cc:348] T 48f4038e5090490580fa9acf38cb7404 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:36.601068 10654 tablet_bootstrap.cc:492] T 48f4038e5090490580fa9acf38cb7404 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:36.601097 10620 raft_consensus.cc:374] T 48f4038e5090490580fa9acf38cb7404 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.601138 10620 raft_consensus.cc:729] T 48f4038e5090490580fa9acf38cb7404 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.601140 10654 ts_tablet_manager.cc:1367] T 48f4038e5090490580fa9acf38cb7404 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:36.601248 10620 consensus_queue.cc:260] T 48f4038e5090490580fa9acf38cb7404 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:36.601531 10620 ts_tablet_manager.cc:1398] T 48f4038e5090490580fa9acf38cb7404 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:36.601579 10654 raft_consensus.cc:348] T 48f4038e5090490580fa9acf38cb7404 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:36.601683 10654 raft_consensus.cc:374] T 48f4038e5090490580fa9acf38cb7404 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.601724 10654 raft_consensus.cc:729] T 48f4038e5090490580fa9acf38cb7404 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.601845 10654 consensus_queue.cc:260] T 48f4038e5090490580fa9acf38cb7404 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:36.602159 10654 ts_tablet_manager.cc:1398] T 48f4038e5090490580fa9acf38cb7404 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:36.603502 10652 tablet_bootstrap.cc:492] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:36.603592 10652 ts_tablet_manager.cc:1367] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:59:36.603976 10652 raft_consensus.cc:348] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:36.604053 10652 raft_consensus.cc:374] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.604072 10652 raft_consensus.cc:729] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.604161 10652 consensus_queue.cc:260] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:36.604386 10652 ts_tablet_manager.cc:1398] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:36.660691 10444 raft_consensus.cc:482] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:36.660816 10444 raft_consensus.cc:504] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:36.661219 10444 leader_election.cc:288] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:36.661648 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f4038e5090490580fa9acf38cb7404" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:36.661715 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f4038e5090490580fa9acf38cb7404" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:36.661787 9211 raft_consensus.cc:2454] T 48f4038e5090490580fa9acf38cb7404 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:36.661883 8947 raft_consensus.cc:2454] T 48f4038e5090490580fa9acf38cb7404 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:36.662024 6726 leader_election.cc:302] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:36.662214 10444 raft_consensus.cc:2781] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:36.662283 10444 raft_consensus.cc:482] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:36.662312 10444 raft_consensus.cc:3037] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.663147 10444 raft_consensus.cc:504] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:36.663522 10444 leader_election.cc:288] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:36.663839 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f4038e5090490580fa9acf38cb7404" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:36.663888 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f4038e5090490580fa9acf38cb7404" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:36.663969 9211 raft_consensus.cc:3037] T 48f4038e5090490580fa9acf38cb7404 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.664000 8947 raft_consensus.cc:3037] T 48f4038e5090490580fa9acf38cb7404 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.664760 8947 raft_consensus.cc:2454] T 48f4038e5090490580fa9acf38cb7404 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:36.664784 9211 raft_consensus.cc:2454] T 48f4038e5090490580fa9acf38cb7404 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:36.665033 6726 leader_election.cc:302] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:36.665206 10444 raft_consensus.cc:2781] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:36.665313 10444 raft_consensus.cc:686] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:36.665454 10444 consensus_queue.cc:237] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:36.667209 6386 catalog_manager.cc:5484] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:36.697497 10637 consensus_queue.cc:1035] T 9358fd0a35c94daabbad58fe99547320 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:36.700404 10673 consensus_queue.cc:1035] T 9358fd0a35c94daabbad58fe99547320 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:36.778837 10673 consensus_queue.cc:1035] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:36.781859 10673 consensus_queue.cc:1035] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:36.782516 6386 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:36.787684 8947 raft_consensus.cc:1261] T 48f4038e5090490580fa9acf38cb7404 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:36.787684 9205 raft_consensus.cc:1261] T 48f4038e5090490580fa9acf38cb7404 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:36.788146 10683 consensus_queue.cc:1035] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:36.788297 10444 consensus_queue.cc:1035] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:36.790882 6386 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet 48f4038e5090490580fa9acf38cb7404 (table rebalancer_dynamic_table [id=3d54da85efa540799bd125c75f2b7a7d]). Expected version 1 got 0
I20240503 13:59:36.791131 10686 tablet.cc:1722] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:36.791268 6379 catalog_manager.cc:5576] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091) does not have the latest schema for tablet 48f4038e5090490580fa9acf38cb7404 (table rebalancer_dynamic_table [id=3d54da85efa540799bd125c75f2b7a7d]). Expected version 1 got 0
I20240503 13:59:36.791524 10686 tablet.cc:1620] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64: MemRowSet was empty: no flush needed.
I20240503 13:59:36.792974 10646 tablet.cc:1722] T 48f4038e5090490580fa9acf38cb7404 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:36.793303 10646 tablet.cc:1620] T 48f4038e5090490580fa9acf38cb7404 P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:36.794620 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=3d54da85efa540799bd125c75f2b7a7d] alter complete (version 1)
I20240503 13:59:36.797897 6386 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet 48f4038e5090490580fa9acf38cb7404 (table rebalancer_dynamic_table [id=3d54da85efa540799bd125c75f2b7a7d]). Expected version 1 got 0
I20240503 13:59:36.799074 10687 tablet.cc:1722] T 48f4038e5090490580fa9acf38cb7404 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:36.799340 10687 tablet.cc:1620] T 48f4038e5090490580fa9acf38cb7404 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:36.899356 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:36.904785 10686 tablet.cc:1722] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:36.905225 10646 tablet.cc:1722] T 48f4038e5090490580fa9acf38cb7404 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:36.905192 10686 tablet.cc:1620] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64: MemRowSet was empty: no flush needed.
I20240503 13:59:36.905495 10646 tablet.cc:1620] T 48f4038e5090490580fa9acf38cb7404 P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:36.907117 10687 tablet.cc:1722] T 48f4038e5090490580fa9acf38cb7404 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:36.907434 10687 tablet.cc:1620] T 48f4038e5090490580fa9acf38cb7404 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:36.910094 10304 consensus_queue.cc:1035] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:36.910784 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=3d54da85efa540799bd125c75f2b7a7d] alter complete (version 2)
I20240503 13:59:36.913530 10645 consensus_queue.cc:1035] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:37.011878 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:37.017237 10686 tablet.cc:1722] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:37.017715 10687 tablet.cc:1722] T 48f4038e5090490580fa9acf38cb7404 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:37.019564 10646 tablet.cc:1722] T 48f4038e5090490580fa9acf38cb7404 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:37.021014 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:37.022769 6386 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=3d54da85efa540799bd125c75f2b7a7d] alter complete (version 3)
I20240503 13:59:37.027240 9191 tablet_service.cc:1467] Processing CreateTablet for tablet d588a730a0c94d4f9446049e537c6727 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=a8e55189edbc4c54bebc8aac90a27231]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:37.027539 9191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d588a730a0c94d4f9446049e537c6727. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:37.027552 9322 tablet_service.cc:1467] Processing CreateTablet for tablet 1be8960eed8148f5958f6bb412c30a7e (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=a8e55189edbc4c54bebc8aac90a27231]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:37.027582 6508 tablet_service.cc:1467] Processing CreateTablet for tablet f1291c3100bb4533891a1a7a3357d363 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=a8e55189edbc4c54bebc8aac90a27231]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:37.027782 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be8960eed8148f5958f6bb412c30a7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:37.027822 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1291c3100bb4533891a1a7a3357d363. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:37.029531 10691 tablet_bootstrap.cc:492] T f1291c3100bb4533891a1a7a3357d363 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:37.029748 6640 tablet_service.cc:1467] Processing CreateTablet for tablet d588a730a0c94d4f9446049e537c6727 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=a8e55189edbc4c54bebc8aac90a27231]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:37.029848 9321 tablet_service.cc:1467] Processing CreateTablet for tablet f1291c3100bb4533891a1a7a3357d363 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=a8e55189edbc4c54bebc8aac90a27231]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:37.029999 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d588a730a0c94d4f9446049e537c6727. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:37.030061 9321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1291c3100bb4533891a1a7a3357d363. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:37.030735 10692 tablet_bootstrap.cc:492] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:37.030804 10691 tablet_bootstrap.cc:654] T f1291c3100bb4533891a1a7a3357d363 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:37.031689 10692 tablet_bootstrap.cc:654] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:37.032250 10620 tablet_bootstrap.cc:492] T d588a730a0c94d4f9446049e537c6727 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:37.032713 10694 tablet_bootstrap.cc:492] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:37.033118 10692 tablet_bootstrap.cc:492] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:37.033169 10620 tablet_bootstrap.cc:654] T d588a730a0c94d4f9446049e537c6727 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:37.033203 10692 ts_tablet_manager.cc:1367] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:37.033576 10692 raft_consensus.cc:348] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:37.033689 10692 raft_consensus.cc:374] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:37.033880 10692 raft_consensus.cc:729] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:37.034201 9061 tablet_service.cc:1467] Processing CreateTablet for tablet d588a730a0c94d4f9446049e537c6727 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=a8e55189edbc4c54bebc8aac90a27231]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:37.034195 9060 tablet_service.cc:1467] Processing CreateTablet for tablet f1291c3100bb4533891a1a7a3357d363 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=a8e55189edbc4c54bebc8aac90a27231]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:37.034171 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 1be8960eed8148f5958f6bb412c30a7e (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=a8e55189edbc4c54bebc8aac90a27231]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:37.034507 10694 tablet_bootstrap.cc:654] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:37.034518 9060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1291c3100bb4533891a1a7a3357d363. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:37.034638 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be8960eed8148f5958f6bb412c30a7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:37.034646 6769 tablet_service.cc:1467] Processing CreateTablet for tablet 1be8960eed8148f5958f6bb412c30a7e (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=a8e55189edbc4c54bebc8aac90a27231]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:37.034905 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be8960eed8148f5958f6bb412c30a7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:37.037451 10691 tablet_bootstrap.cc:492] T f1291c3100bb4533891a1a7a3357d363 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:37.037518 10691 ts_tablet_manager.cc:1367] T f1291c3100bb4533891a1a7a3357d363 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:59:37.037631 10652 tablet_bootstrap.cc:492] T 1be8960eed8148f5958f6bb412c30a7e P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:37.037707 10694 tablet_bootstrap.cc:492] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:37.037777 10694 ts_tablet_manager.cc:1367] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:37.037961 10691 raft_consensus.cc:348] T f1291c3100bb4533891a1a7a3357d363 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:37.038096 10691 raft_consensus.cc:374] T f1291c3100bb4533891a1a7a3357d363 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:37.038120 10691 raft_consensus.cc:729] T f1291c3100bb4533891a1a7a3357d363 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:37.038241 10691 consensus_queue.cc:260] T f1291c3100bb4533891a1a7a3357d363 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:37.038295 10694 raft_consensus.cc:348] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:37.038446 10694 raft_consensus.cc:374] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:37.038488 10694 raft_consensus.cc:729] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:37.038568 10691 ts_tablet_manager.cc:1398] T f1291c3100bb4533891a1a7a3357d363 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:37.038594 10652 tablet_bootstrap.cc:654] T 1be8960eed8148f5958f6bb412c30a7e P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:37.038635 10694 consensus_queue.cc:260] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:37.038972 10694 ts_tablet_manager.cc:1398] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:37.040411 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d588a730a0c94d4f9446049e537c6727. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:37.044173 10652 tablet_bootstrap.cc:492] T 1be8960eed8148f5958f6bb412c30a7e P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:37.044247 10652 ts_tablet_manager.cc:1367] T 1be8960eed8148f5958f6bb412c30a7e P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:59:37.045179 10652 raft_consensus.cc:348] T 1be8960eed8148f5958f6bb412c30a7e P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:37.045277 10652 raft_consensus.cc:374] T 1be8960eed8148f5958f6bb412c30a7e P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:37.045315 10652 raft_consensus.cc:729] T 1be8960eed8148f5958f6bb412c30a7e P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:37.045444 10652 consensus_queue.cc:260] T 1be8960eed8148f5958f6bb412c30a7e P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:37.045743 10652 ts_tablet_manager.cc:1398] T 1be8960eed8148f5958f6bb412c30a7e P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:37.046664 10692 consensus_queue.cc:260] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:37.047153 10692 ts_tablet_manager.cc:1398] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240503 13:59:37.047395 10698 tablet_bootstrap.cc:492] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:37.047485 10692 tablet_bootstrap.cc:492] T f1291c3100bb4533891a1a7a3357d363 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:37.048430 10698 tablet_bootstrap.cc:654] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:37.048625 10692 tablet_bootstrap.cc:654] T f1291c3100bb4533891a1a7a3357d363 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:37.049474 10698 tablet_bootstrap.cc:492] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:37.049603 10698 ts_tablet_manager.cc:1367] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:37.049806 10692 tablet_bootstrap.cc:492] T f1291c3100bb4533891a1a7a3357d363 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:37.049888 10692 ts_tablet_manager.cc:1367] T f1291c3100bb4533891a1a7a3357d363 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:37.050081 10698 raft_consensus.cc:348] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:37.050158 10698 raft_consensus.cc:374] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:37.050177 10698 raft_consensus.cc:729] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:37.050276 10698 consensus_queue.cc:260] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:37.050585 10698 ts_tablet_manager.cc:1398] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:37.050571 10692 raft_consensus.cc:348] T f1291c3100bb4533891a1a7a3357d363 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:37.050729 10692 raft_consensus.cc:374] T f1291c3100bb4533891a1a7a3357d363 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:37.050803 10692 raft_consensus.cc:729] T f1291c3100bb4533891a1a7a3357d363 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:37.050962 10692 consensus_queue.cc:260] T f1291c3100bb4533891a1a7a3357d363 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:37.051265 10692 ts_tablet_manager.cc:1398] T f1291c3100bb4533891a1a7a3357d363 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:37.051409 10698 tablet_bootstrap.cc:492] T d588a730a0c94d4f9446049e537c6727 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:37.052321 10698 tablet_bootstrap.cc:654] T d588a730a0c94d4f9446049e537c6727 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:37.054162 10698 tablet_bootstrap.cc:492] T d588a730a0c94d4f9446049e537c6727 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:37.054291 10698 ts_tablet_manager.cc:1367] T d588a730a0c94d4f9446049e537c6727 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:37.054713 10698 raft_consensus.cc:348] T d588a730a0c94d4f9446049e537c6727 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:37.054868 10698 raft_consensus.cc:374] T d588a730a0c94d4f9446049e537c6727 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:37.054965 10698 raft_consensus.cc:729] T d588a730a0c94d4f9446049e537c6727 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:37.055143 10698 consensus_queue.cc:260] T d588a730a0c94d4f9446049e537c6727 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:37.055440 10698 ts_tablet_manager.cc:1398] T d588a730a0c94d4f9446049e537c6727 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:37.055639 10620 tablet_bootstrap.cc:492] T d588a730a0c94d4f9446049e537c6727 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:37.055691 10698 tablet_bootstrap.cc:492] T 1be8960eed8148f5958f6bb412c30a7e P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:37.055759 10620 ts_tablet_manager.cc:1367] T d588a730a0c94d4f9446049e537c6727 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240503 13:59:37.056252 10620 raft_consensus.cc:348] T d588a730a0c94d4f9446049e537c6727 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:37.056363 10620 raft_consensus.cc:374] T d588a730a0c94d4f9446049e537c6727 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:37.056393 10620 raft_consensus.cc:729] T d588a730a0c94d4f9446049e537c6727 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:37.056528 10620 consensus_queue.cc:260] T d588a730a0c94d4f9446049e537c6727 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:37.056824 10620 ts_tablet_manager.cc:1398] T d588a730a0c94d4f9446049e537c6727 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:37.058033 10698 tablet_bootstrap.cc:654] T 1be8960eed8148f5958f6bb412c30a7e P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:37.059365 10698 tablet_bootstrap.cc:492] T 1be8960eed8148f5958f6bb412c30a7e P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:37.059449 10698 ts_tablet_manager.cc:1367] T 1be8960eed8148f5958f6bb412c30a7e P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:37.059887 10698 raft_consensus.cc:348] T 1be8960eed8148f5958f6bb412c30a7e P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:37.059967 10698 raft_consensus.cc:374] T 1be8960eed8148f5958f6bb412c30a7e P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:37.059986 10698 raft_consensus.cc:729] T 1be8960eed8148f5958f6bb412c30a7e P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:37.060092 10698 consensus_queue.cc:260] T 1be8960eed8148f5958f6bb412c30a7e P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:37.060326 10698 ts_tablet_manager.cc:1398] T 1be8960eed8148f5958f6bb412c30a7e P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:37.073652 10624 raft_consensus.cc:482] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:37.073763 10624 raft_consensus.cc:504] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:37.074203 10624 leader_election.cc:288] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:37.074519 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1291c3100bb4533891a1a7a3357d363" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:37.074615 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1291c3100bb4533891a1a7a3357d363" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:37.074664 6527 raft_consensus.cc:2454] T f1291c3100bb4533891a1a7a3357d363 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:37.074748 9343 raft_consensus.cc:2454] T f1291c3100bb4533891a1a7a3357d363 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:37.074950 9016 leader_election.cc:302] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:37.075116 10624 raft_consensus.cc:2781] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:37.075182 10624 raft_consensus.cc:482] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:37.075225 10624 raft_consensus.cc:3037] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:37.076336 10624 raft_consensus.cc:504] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:37.076648 10624 leader_election.cc:288] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:37.076959 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1291c3100bb4533891a1a7a3357d363" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:37.077056 9343 raft_consensus.cc:3037] T f1291c3100bb4533891a1a7a3357d363 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:37.077701 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1291c3100bb4533891a1a7a3357d363" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:37.077807 6527 raft_consensus.cc:3037] T f1291c3100bb4533891a1a7a3357d363 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:37.077791 9343 raft_consensus.cc:2454] T f1291c3100bb4533891a1a7a3357d363 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:37.078143 9017 leader_election.cc:302] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:37.078326 10624 raft_consensus.cc:2781] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:37.078424 10624 raft_consensus.cc:686] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 13:59:37.078560 10624 consensus_queue.cc:237] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:37.078831 6527 raft_consensus.cc:2454] T f1291c3100bb4533891a1a7a3357d363 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:37.080623 6386 catalog_manager.cc:5484] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:37.096504 10605 raft_consensus.cc:482] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:37.096601 10605 raft_consensus.cc:504] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:37.097025 10605 leader_election.cc:288] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:37.097357 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d588a730a0c94d4f9446049e537c6727" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:37.097364 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d588a730a0c94d4f9446049e537c6727" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:37.097493 9205 raft_consensus.cc:2454] T d588a730a0c94d4f9446049e537c6727 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:37.097489 9081 raft_consensus.cc:2454] T d588a730a0c94d4f9446049e537c6727 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:37.097793 6595 leader_election.cc:302] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:37.098065 10605 raft_consensus.cc:2781] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:37.098120 10605 raft_consensus.cc:482] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:37.098146 10605 raft_consensus.cc:3037] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:37.099052 10605 raft_consensus.cc:504] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:37.099370 10605 leader_election.cc:288] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:37.099771 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d588a730a0c94d4f9446049e537c6727" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:37.099857 9081 raft_consensus.cc:3037] T d588a730a0c94d4f9446049e537c6727 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:37.099807 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d588a730a0c94d4f9446049e537c6727" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:37.099943 9205 raft_consensus.cc:3037] T d588a730a0c94d4f9446049e537c6727 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:37.100566 9081 raft_consensus.cc:2454] T d588a730a0c94d4f9446049e537c6727 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:37.100709 9205 raft_consensus.cc:2454] T d588a730a0c94d4f9446049e537c6727 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:37.100831 6595 leader_election.cc:302] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:37.100991 10605 raft_consensus.cc:2781] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:37.101095 10605 raft_consensus.cc:686] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 13:59:37.101235 10605 consensus_queue.cc:237] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:37.103076 6386 catalog_manager.cc:5484] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:37.126744 6386 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:37.126901 6386 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:37.129160 6386 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 48f4038e5090490580fa9acf38cb7404
I20240503 13:59:37.129638 9191 tablet_service.cc:1514] Processing DeleteTablet for tablet 48f4038e5090490580fa9acf38cb7404 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:37 UTC) from {username='slave'} at 127.0.0.1:57960
I20240503 13:59:37.129717 8926 tablet_service.cc:1514] Processing DeleteTablet for tablet 48f4038e5090490580fa9acf38cb7404 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:37 UTC) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:37.129717 6769 tablet_service.cc:1514] Processing DeleteTablet for tablet 48f4038e5090490580fa9acf38cb7404 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:37 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:37.130177 10706 tablet_replica.cc:330] T 48f4038e5090490580fa9acf38cb7404 P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 13:59:37.130311 10707 tablet_replica.cc:330] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:37.130326 10706 raft_consensus.cc:2229] T 48f4038e5090490580fa9acf38cb7404 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:37.130466 10707 raft_consensus.cc:2229] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:37.130498 10706 raft_consensus.cc:2258] T 48f4038e5090490580fa9acf38cb7404 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:37.130690 10707 raft_consensus.cc:2258] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:37.130930 10708 tablet_replica.cc:330] T 48f4038e5090490580fa9acf38cb7404 P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:37.131112 10708 raft_consensus.cc:2229] T 48f4038e5090490580fa9acf38cb7404 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:37.131155 10706 ts_tablet_manager.cc:1875] T 48f4038e5090490580fa9acf38cb7404 P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:37.131247 10708 raft_consensus.cc:2258] T 48f4038e5090490580fa9acf38cb7404 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:37.131335 10707 ts_tablet_manager.cc:1875] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:37.131840 10708 ts_tablet_manager.cc:1875] T 48f4038e5090490580fa9acf38cb7404 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:37.133267 10706 ts_tablet_manager.cc:1888] T 48f4038e5090490580fa9acf38cb7404 P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:37.133338 10706 log.cc:1196] T 48f4038e5090490580fa9acf38cb7404 P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/48f4038e5090490580fa9acf38cb7404
I20240503 13:59:37.133730 10706 ts_tablet_manager.cc:1909] T 48f4038e5090490580fa9acf38cb7404 P ab1d21935d6049888eb0ceccc3c9d735: Deleting consensus metadata
I20240503 13:59:37.133934 10707 ts_tablet_manager.cc:1888] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:37.133929 10708 ts_tablet_manager.cc:1888] T 48f4038e5090490580fa9acf38cb7404 P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:37.134007 10708 log.cc:1196] T 48f4038e5090490580fa9acf38cb7404 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/48f4038e5090490580fa9acf38cb7404
I20240503 13:59:37.134007 10707 log.cc:1196] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/48f4038e5090490580fa9acf38cb7404
I20240503 13:59:37.134301 10708 ts_tablet_manager.cc:1909] T 48f4038e5090490580fa9acf38cb7404 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting consensus metadata
I20240503 13:59:37.134326 10707 ts_tablet_manager.cc:1909] T 48f4038e5090490580fa9acf38cb7404 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:37.134367 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet 48f4038e5090490580fa9acf38cb7404 (table rebalancer_dynamic_table_new_name [id=3d54da85efa540799bd125c75f2b7a7d]) successfully deleted
I20240503 13:59:37.134953 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet 48f4038e5090490580fa9acf38cb7404 (table rebalancer_dynamic_table_new_name [id=3d54da85efa540799bd125c75f2b7a7d]) successfully deleted
I20240503 13:59:37.135093 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 48f4038e5090490580fa9acf38cb7404 (table rebalancer_dynamic_table_new_name [id=3d54da85efa540799bd125c75f2b7a7d]) successfully deleted
I20240503 13:59:37.144584 10564 raft_consensus.cc:482] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:37.144681 10564 raft_consensus.cc:504] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:37.145113 10564 leader_election.cc:288] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:37.145416 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be8960eed8148f5958f6bb412c30a7e" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:37.145480 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be8960eed8148f5958f6bb412c30a7e" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:37.145550 9081 raft_consensus.cc:2454] T 1be8960eed8148f5958f6bb412c30a7e P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:37.145609 6791 raft_consensus.cc:2454] T 1be8960eed8148f5958f6bb412c30a7e P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:37.145838 9278 leader_election.cc:302] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:37.146028 10564 raft_consensus.cc:2781] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:37.146090 10564 raft_consensus.cc:482] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:37.146121 10564 raft_consensus.cc:3037] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:37.146953 10564 raft_consensus.cc:504] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:37.147357 10564 leader_election.cc:288] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:37.147578 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be8960eed8148f5958f6bb412c30a7e" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:37.147694 9081 raft_consensus.cc:3037] T 1be8960eed8148f5958f6bb412c30a7e P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:37.147666 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be8960eed8148f5958f6bb412c30a7e" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:37.147775 6791 raft_consensus.cc:3037] T 1be8960eed8148f5958f6bb412c30a7e P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:37.148452 9081 raft_consensus.cc:2454] T 1be8960eed8148f5958f6bb412c30a7e P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:37.148546 6791 raft_consensus.cc:2454] T 1be8960eed8148f5958f6bb412c30a7e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:37.148964 9278 leader_election.cc:302] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:37.149137 10564 raft_consensus.cc:2781] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:37.149220 10564 raft_consensus.cc:686] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:37.149363 10564 consensus_queue.cc:237] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:37.151500 6386 catalog_manager.cc:5484] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:37.231006 6379 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:37.236451 6769 tablet_service.cc:1467] Processing CreateTablet for tablet b64360e19010440e946a4bb8f564610d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=568d7289694346268a4fc7e5d46a06ff]), partition=
I20240503 13:59:37.236706 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b64360e19010440e946a4bb8f564610d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:37.236713 6902 tablet_service.cc:1467] Processing CreateTablet for tablet b64360e19010440e946a4bb8f564610d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=568d7289694346268a4fc7e5d46a06ff]), partition=
I20240503 13:59:37.236938 6902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b64360e19010440e946a4bb8f564610d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:37.238597 10652 tablet_bootstrap.cc:492] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:37.239109 9058 tablet_service.cc:1467] Processing CreateTablet for tablet b64360e19010440e946a4bb8f564610d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=568d7289694346268a4fc7e5d46a06ff]), partition=
I20240503 13:59:37.239382 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b64360e19010440e946a4bb8f564610d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:37.239506 10652 tablet_bootstrap.cc:654] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:37.241155 10652 tablet_bootstrap.cc:492] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:37.241238 10652 ts_tablet_manager.cc:1367] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:37.241423 10698 tablet_bootstrap.cc:492] T b64360e19010440e946a4bb8f564610d P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:37.241779 10652 raft_consensus.cc:348] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:37.241894 10652 raft_consensus.cc:374] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:37.241925 10652 raft_consensus.cc:729] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:37.242035 10652 consensus_queue.cc:260] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:37.242172 10698 tablet_bootstrap.cc:654] T b64360e19010440e946a4bb8f564610d P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:37.242312 10652 ts_tablet_manager.cc:1398] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:37.243263 10710 tablet_bootstrap.cc:492] T b64360e19010440e946a4bb8f564610d P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:37.243438 10698 tablet_bootstrap.cc:492] T b64360e19010440e946a4bb8f564610d P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:37.243520 10698 ts_tablet_manager.cc:1367] T b64360e19010440e946a4bb8f564610d P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:37.243873 10698 raft_consensus.cc:348] T b64360e19010440e946a4bb8f564610d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:37.243968 10698 raft_consensus.cc:374] T b64360e19010440e946a4bb8f564610d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:37.244004 10698 raft_consensus.cc:729] T b64360e19010440e946a4bb8f564610d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:37.244138 10698 consensus_queue.cc:260] T b64360e19010440e946a4bb8f564610d P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:37.244208 10710 tablet_bootstrap.cc:654] T b64360e19010440e946a4bb8f564610d P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:37.244380 10698 ts_tablet_manager.cc:1398] T b64360e19010440e946a4bb8f564610d P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:37.245314 10710 tablet_bootstrap.cc:492] T b64360e19010440e946a4bb8f564610d P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:37.245407 10710 ts_tablet_manager.cc:1367] T b64360e19010440e946a4bb8f564610d P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:37.245754 10710 raft_consensus.cc:348] T b64360e19010440e946a4bb8f564610d P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:37.245862 10710 raft_consensus.cc:374] T b64360e19010440e946a4bb8f564610d P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:37.245916 10710 raft_consensus.cc:729] T b64360e19010440e946a4bb8f564610d P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:37.246050 10710 consensus_queue.cc:260] T b64360e19010440e946a4bb8f564610d P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:37.246340 10710 ts_tablet_manager.cc:1398] T b64360e19010440e946a4bb8f564610d P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:37.522019 10605 consensus_queue.cc:1035] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:37.525949 10673 consensus_queue.cc:1035] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:37.526332 10716 consensus_queue.cc:1035] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 13:59:37.530516 10673 consensus_queue.cc:1035] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:37.548300 10683 raft_consensus.cc:482] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:37.548414 10683 raft_consensus.cc:504] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:37.548827 10683 leader_election.cc:288] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:37.549216 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b64360e19010440e946a4bb8f564610d" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:37.549237 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b64360e19010440e946a4bb8f564610d" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:37.549350 9080 raft_consensus.cc:2454] T b64360e19010440e946a4bb8f564610d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:37.549402 6920 raft_consensus.cc:2454] T b64360e19010440e946a4bb8f564610d P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:37.549636 6726 leader_election.cc:302] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:37.549829 10683 raft_consensus.cc:2781] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:37.549902 10683 raft_consensus.cc:482] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:37.549955 10683 raft_consensus.cc:3037] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:37.550886 10683 raft_consensus.cc:504] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:37.551200 10683 leader_election.cc:288] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:37.551687 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b64360e19010440e946a4bb8f564610d" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:37.551811 6920 raft_consensus.cc:3037] T b64360e19010440e946a4bb8f564610d P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:37.551826 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b64360e19010440e946a4bb8f564610d" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:37.551923 9080 raft_consensus.cc:3037] T b64360e19010440e946a4bb8f564610d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:37.552872 6920 raft_consensus.cc:2454] T b64360e19010440e946a4bb8f564610d P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:37.552882 9080 raft_consensus.cc:2454] T b64360e19010440e946a4bb8f564610d P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:37.553117 6726 leader_election.cc:302] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:37.553285 10683 raft_consensus.cc:2781] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:37.553383 10683 raft_consensus.cc:686] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:37.553519 10683 consensus_queue.cc:237] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:37.555678 6386 catalog_manager.cc:5484] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:37.668689 6386 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:37.674870 8926 tablet_service.cc:1467] Processing CreateTablet for tablet a914c22ceff44fc685aad0fdcd04d844 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8e3c65bd2d8c4b5595ccf6d282f48a82]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:37.674870 6902 tablet_service.cc:1467] Processing CreateTablet for tablet a914c22ceff44fc685aad0fdcd04d844 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8e3c65bd2d8c4b5595ccf6d282f48a82]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:37.675184 8926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a914c22ceff44fc685aad0fdcd04d844. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:37.675187 6902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a914c22ceff44fc685aad0fdcd04d844. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:37.675549 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 82f1b400e34743db819a831b73365e00 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8e3c65bd2d8c4b5595ccf6d282f48a82]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:37.675765 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82f1b400e34743db819a831b73365e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:37.675784 6769 tablet_service.cc:1467] Processing CreateTablet for tablet 96ce36eb7c25413eaae380f32b465df7 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8e3c65bd2d8c4b5595ccf6d282f48a82]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:37.676016 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ce36eb7c25413eaae380f32b465df7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:37.677568 10727 tablet_bootstrap.cc:492] T a914c22ceff44fc685aad0fdcd04d844 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:37.677837 10652 tablet_bootstrap.cc:492] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:37.678516 10727 tablet_bootstrap.cc:654] T a914c22ceff44fc685aad0fdcd04d844 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:37.678834 10652 tablet_bootstrap.cc:654] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:37.679896 6639 tablet_service.cc:1467] Processing CreateTablet for tablet 96ce36eb7c25413eaae380f32b465df7 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8e3c65bd2d8c4b5595ccf6d282f48a82]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:37.679872 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 82f1b400e34743db819a831b73365e00 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8e3c65bd2d8c4b5595ccf6d282f48a82]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:37.680142 10727 tablet_bootstrap.cc:492] T a914c22ceff44fc685aad0fdcd04d844 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:37.680166 6639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ce36eb7c25413eaae380f32b465df7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:37.680219 10727 ts_tablet_manager.cc:1367] T a914c22ceff44fc685aad0fdcd04d844 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:37.680217 10710 tablet_bootstrap.cc:492] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:37.680637 10727 raft_consensus.cc:348] T a914c22ceff44fc685aad0fdcd04d844 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:37.680733 10727 raft_consensus.cc:374] T a914c22ceff44fc685aad0fdcd04d844 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:37.680760 10727 raft_consensus.cc:729] T a914c22ceff44fc685aad0fdcd04d844 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:37.680737 9321 tablet_service.cc:1467] Processing CreateTablet for tablet a914c22ceff44fc685aad0fdcd04d844 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8e3c65bd2d8c4b5595ccf6d282f48a82]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:37.680866 10727 consensus_queue.cc:260] T a914c22ceff44fc685aad0fdcd04d844 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:37.681015 9321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a914c22ceff44fc685aad0fdcd04d844. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:37.681138 10727 ts_tablet_manager.cc:1398] T a914c22ceff44fc685aad0fdcd04d844 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:37.681243 10710 tablet_bootstrap.cc:654] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:37.681308 10727 tablet_bootstrap.cc:492] T 82f1b400e34743db819a831b73365e00 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:37.682123 10727 tablet_bootstrap.cc:654] T 82f1b400e34743db819a831b73365e00 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:37.682646 9322 tablet_service.cc:1467] Processing CreateTablet for tablet 96ce36eb7c25413eaae380f32b465df7 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8e3c65bd2d8c4b5595ccf6d282f48a82]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:37.682749 10710 tablet_bootstrap.cc:492] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:37.682828 10710 ts_tablet_manager.cc:1367] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:59:37.682897 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ce36eb7c25413eaae380f32b465df7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:37.683339 10734 tablet_bootstrap.cc:492] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:37.683272 10710 raft_consensus.cc:348] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:37.683396 10710 raft_consensus.cc:374] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:37.683431 10710 raft_consensus.cc:729] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:37.683526 10710 consensus_queue.cc:260] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:37.683785 10710 ts_tablet_manager.cc:1398] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:37.684303 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82f1b400e34743db819a831b73365e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:37.684350 10734 tablet_bootstrap.cc:654] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:37.685722 10734 tablet_bootstrap.cc:492] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:37.685797 10734 ts_tablet_manager.cc:1367] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:37.686235 10734 raft_consensus.cc:348] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:37.686326 10734 raft_consensus.cc:374] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:37.686376 10734 raft_consensus.cc:729] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:37.686542 10734 consensus_queue.cc:260] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:37.686724 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 82f1b400e34743db819a831b73365e00 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8e3c65bd2d8c4b5595ccf6d282f48a82]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:37.686980 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82f1b400e34743db819a831b73365e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:37.687681 10734 ts_tablet_manager.cc:1398] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:37.687829 10734 tablet_bootstrap.cc:492] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:37.688318 10652 tablet_bootstrap.cc:492] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:37.688413 10652 ts_tablet_manager.cc:1367] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 13:59:37.688685 10734 tablet_bootstrap.cc:654] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:37.688829 10652 raft_consensus.cc:348] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:37.688943 10652 raft_consensus.cc:374] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:37.688975 10652 raft_consensus.cc:729] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:37.689079 10652 consensus_queue.cc:260] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:37.689410 10652 ts_tablet_manager.cc:1398] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:37.689776 10736 tablet_bootstrap.cc:492] T 82f1b400e34743db819a831b73365e00 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:37.690332 10734 tablet_bootstrap.cc:492] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:37.690403 10734 ts_tablet_manager.cc:1367] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:37.690725 10736 tablet_bootstrap.cc:654] T 82f1b400e34743db819a831b73365e00 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:37.690800 10734 raft_consensus.cc:348] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:37.690894 10734 raft_consensus.cc:374] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:37.690924 10734 raft_consensus.cc:729] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:37.691062 10734 consensus_queue.cc:260] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:37.691357 10734 ts_tablet_manager.cc:1398] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:37.694831 10732 tablet_bootstrap.cc:492] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:37.695705 10732 tablet_bootstrap.cc:654] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:37.695914 10727 tablet_bootstrap.cc:492] T 82f1b400e34743db819a831b73365e00 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:37.696003 10727 ts_tablet_manager.cc:1367] T 82f1b400e34743db819a831b73365e00 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 13:59:37.696463 10736 tablet_bootstrap.cc:492] T 82f1b400e34743db819a831b73365e00 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:37.696455 10727 raft_consensus.cc:348] T 82f1b400e34743db819a831b73365e00 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:37.696543 10736 ts_tablet_manager.cc:1367] T 82f1b400e34743db819a831b73365e00 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:59:37.696570 10727 raft_consensus.cc:374] T 82f1b400e34743db819a831b73365e00 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:37.696624 10727 raft_consensus.cc:729] T 82f1b400e34743db819a831b73365e00 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:37.696750 10727 consensus_queue.cc:260] T 82f1b400e34743db819a831b73365e00 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:37.697047 10727 ts_tablet_manager.cc:1398] T 82f1b400e34743db819a831b73365e00 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:37.697301 10732 tablet_bootstrap.cc:492] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:37.697400 10732 ts_tablet_manager.cc:1367] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:37.697646 10736 raft_consensus.cc:348] T 82f1b400e34743db819a831b73365e00 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:37.697896 10736 raft_consensus.cc:374] T 82f1b400e34743db819a831b73365e00 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:37.697954 10736 raft_consensus.cc:729] T 82f1b400e34743db819a831b73365e00 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:37.698117 10736 consensus_queue.cc:260] T 82f1b400e34743db819a831b73365e00 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:37.698433 10736 ts_tablet_manager.cc:1398] T 82f1b400e34743db819a831b73365e00 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:37.698695 10732 raft_consensus.cc:348] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:37.698784 10732 raft_consensus.cc:374] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:37.698818 10732 raft_consensus.cc:729] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:37.698961 10732 consensus_queue.cc:260] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:37.699240 10732 ts_tablet_manager.cc:1398] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:37.700165 10732 tablet_bootstrap.cc:492] T 82f1b400e34743db819a831b73365e00 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:37.703027 10732 tablet_bootstrap.cc:654] T 82f1b400e34743db819a831b73365e00 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:37.704367 10732 tablet_bootstrap.cc:492] T 82f1b400e34743db819a831b73365e00 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:37.704455 10732 ts_tablet_manager.cc:1367] T 82f1b400e34743db819a831b73365e00 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 13:59:37.704668 6383 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:37.705009 10732 raft_consensus.cc:348] T 82f1b400e34743db819a831b73365e00 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:37.705104 10732 raft_consensus.cc:374] T 82f1b400e34743db819a831b73365e00 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:37.705140 10732 raft_consensus.cc:729] T 82f1b400e34743db819a831b73365e00 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:37.705535 10732 consensus_queue.cc:260] T 82f1b400e34743db819a831b73365e00 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:37.706692 10732 ts_tablet_manager.cc:1398] T 82f1b400e34743db819a831b73365e00 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:37.709125 6920 raft_consensus.cc:1261] T b64360e19010440e946a4bb8f564610d P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:37.709157 9080 raft_consensus.cc:1261] T b64360e19010440e946a4bb8f564610d P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:37.709551 10663 consensus_queue.cc:1035] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:37.709702 10683 consensus_queue.cc:1035] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:37.712493 10746 tablet.cc:1722] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:37.712874 10746 tablet.cc:1620] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64: MemRowSet was empty: no flush needed.
I20240503 13:59:37.712847 6383 catalog_manager.cc:5576] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531) does not have the latest schema for tablet b64360e19010440e946a4bb8f564610d (table rebalancer_dynamic_table [id=568d7289694346268a4fc7e5d46a06ff]). Expected version 1 got 0
I20240503 13:59:37.713172 10747 tablet.cc:1722] T b64360e19010440e946a4bb8f564610d P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:37.713759 10747 tablet.cc:1620] T b64360e19010440e946a4bb8f564610d P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:37.714222 6379 catalog_manager.cc:5576] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481) does not have the latest schema for tablet b64360e19010440e946a4bb8f564610d (table rebalancer_dynamic_table [id=568d7289694346268a4fc7e5d46a06ff]). Expected version 1 got 0
I20240503 13:59:37.716686 10748 tablet.cc:1722] T b64360e19010440e946a4bb8f564610d P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:37.716974 10748 tablet.cc:1620] T b64360e19010440e946a4bb8f564610d P f58065db70b64ef8b1dc499aa6ae4460: MemRowSet was empty: no flush needed.
I20240503 13:59:37.717407 6386 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=568d7289694346268a4fc7e5d46a06ff] alter complete (version 1)
I20240503 13:59:37.719046 6379 catalog_manager.cc:5576] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531) does not have the latest schema for tablet b64360e19010440e946a4bb8f564610d (table rebalancer_dynamic_table [id=568d7289694346268a4fc7e5d46a06ff]). Expected version 1 got 0
I20240503 13:59:37.731627 10716 raft_consensus.cc:482] T 82f1b400e34743db819a831b73365e00 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:37.731747 10716 raft_consensus.cc:504] T 82f1b400e34743db819a831b73365e00 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:37.732239 10716 leader_election.cc:288] T 82f1b400e34743db819a831b73365e00 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:37.732604 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82f1b400e34743db819a831b73365e00" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:37.732753 8947 raft_consensus.cc:2454] T 82f1b400e34743db819a831b73365e00 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:37.733150 6595 leader_election.cc:302] T 82f1b400e34743db819a831b73365e00 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 2145d15ab2d74c0287839266f1411a43; no voters:
I20240503 13:59:37.733117 10564 consensus_queue.cc:1035] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:37.733273 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82f1b400e34743db819a831b73365e00" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:37.733405 6527 raft_consensus.cc:2454] T 82f1b400e34743db819a831b73365e00 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:37.736523 10564 consensus_queue.cc:1035] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:37.739900 10716 raft_consensus.cc:2781] T 82f1b400e34743db819a831b73365e00 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:37.739966 10716 raft_consensus.cc:482] T 82f1b400e34743db819a831b73365e00 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:37.740006 10716 raft_consensus.cc:3037] T 82f1b400e34743db819a831b73365e00 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:37.741091 10716 raft_consensus.cc:504] T 82f1b400e34743db819a831b73365e00 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:37.741474 10716 leader_election.cc:288] T 82f1b400e34743db819a831b73365e00 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:37.742131 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82f1b400e34743db819a831b73365e00" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:37.742251 6527 raft_consensus.cc:3037] T 82f1b400e34743db819a831b73365e00 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:37.743211 6527 raft_consensus.cc:2454] T 82f1b400e34743db819a831b73365e00 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:37.743472 6595 leader_election.cc:302] T 82f1b400e34743db819a831b73365e00 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:37.743880 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82f1b400e34743db819a831b73365e00" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:37.743937 10716 raft_consensus.cc:2781] T 82f1b400e34743db819a831b73365e00 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:37.744014 8948 raft_consensus.cc:3037] T 82f1b400e34743db819a831b73365e00 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:37.744278 10716 raft_consensus.cc:686] T 82f1b400e34743db819a831b73365e00 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 13:59:37.744452 10716 consensus_queue.cc:237] T 82f1b400e34743db819a831b73365e00 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:37.744967 8948 raft_consensus.cc:2454] T 82f1b400e34743db819a831b73365e00 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:37.746817 6379 catalog_manager.cc:5484] T 82f1b400e34743db819a831b73365e00 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:37.769194 10750 raft_consensus.cc:482] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:37.769310 10750 raft_consensus.cc:504] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:37.769757 10750 leader_election.cc:288] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:37.770128 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ce36eb7c25413eaae380f32b465df7" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:37.770171 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ce36eb7c25413eaae380f32b465df7" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:37.770257 6791 raft_consensus.cc:2454] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:37.770308 6660 raft_consensus.cc:2454] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:37.770551 9277 leader_election.cc:302] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:37.770741 10750 raft_consensus.cc:2781] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:37.770815 10750 raft_consensus.cc:482] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:37.770869 10750 raft_consensus.cc:3037] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:37.771970 10750 raft_consensus.cc:504] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:37.772270 10750 leader_election.cc:288] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:37.772568 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ce36eb7c25413eaae380f32b465df7" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:37.772595 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ce36eb7c25413eaae380f32b465df7" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:37.772681 6660 raft_consensus.cc:3037] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:37.772702 6791 raft_consensus.cc:3037] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:37.773406 6660 raft_consensus.cc:2454] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:37.773437 6791 raft_consensus.cc:2454] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:37.773698 9277 leader_election.cc:302] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:37.773875 10750 raft_consensus.cc:2781] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:37.773948 10750 raft_consensus.cc:686] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:37.774107 10750 consensus_queue.cc:237] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:37.776170 6379 catalog_manager.cc:5484] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } }
W20240503 13:59:37.779515 6707 tablet.cc:2369] T 82f1b400e34743db819a831b73365e00 P 2145d15ab2d74c0287839266f1411a43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:59:37.822194 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:37.828614 10746 tablet.cc:1722] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:37.828909 10746 tablet.cc:1620] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64: MemRowSet was empty: no flush needed.
I20240503 13:59:37.829121 10748 tablet.cc:1722] T b64360e19010440e946a4bb8f564610d P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:37.829411 10748 tablet.cc:1620] T b64360e19010440e946a4bb8f564610d P f58065db70b64ef8b1dc499aa6ae4460: MemRowSet was empty: no flush needed.
W20240503 13:59:37.829770 6576 tablet.cc:2369] T 82f1b400e34743db819a831b73365e00 P 5f46e19644fd4f4b825c86ba6af768da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:59:37.832012 10747 tablet.cc:1722] T b64360e19010440e946a4bb8f564610d P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:37.832293 10747 tablet.cc:1620] T b64360e19010440e946a4bb8f564610d P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:37.833609 10750 raft_consensus.cc:482] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:37.833739 10750 raft_consensus.cc:504] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:37.834106 10750 leader_election.cc:288] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:37.834614 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a914c22ceff44fc685aad0fdcd04d844" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:37.834648 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a914c22ceff44fc685aad0fdcd04d844" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:37.834746 6920 raft_consensus.cc:2454] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:37.834746 8948 raft_consensus.cc:2454] T a914c22ceff44fc685aad0fdcd04d844 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:37.834919 6383 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=568d7289694346268a4fc7e5d46a06ff] alter complete (version 2)
I20240503 13:59:37.835188 9278 leader_election.cc:302] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 0d77752181e944698e809e325e5937e6; no voters:
I20240503 13:59:37.835827 10750 raft_consensus.cc:2781] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:37.835932 10750 raft_consensus.cc:482] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:37.836149 10750 raft_consensus.cc:3037] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:37.837648 10750 raft_consensus.cc:504] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:37.838042 10750 leader_election.cc:288] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:37.838281 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a914c22ceff44fc685aad0fdcd04d844" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:37.838431 6920 raft_consensus.cc:3037] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:37.839742 6920 raft_consensus.cc:2454] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:37.840026 9278 leader_election.cc:302] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:37.840071 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a914c22ceff44fc685aad0fdcd04d844" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:37.840267 8948 raft_consensus.cc:3037] T a914c22ceff44fc685aad0fdcd04d844 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:37.840996 8948 raft_consensus.cc:2454] T a914c22ceff44fc685aad0fdcd04d844 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:37.841187 10750 raft_consensus.cc:2781] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:37.841562 10750 raft_consensus.cc:686] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:37.841702 10750 consensus_queue.cc:237] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:37.843626 6379 catalog_manager.cc:5484] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } }
W20240503 13:59:37.898308 6969 tablet.cc:2369] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:59:37.936909 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:37.943095 10746 tablet.cc:1722] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:37.945698 10747 tablet.cc:1722] T b64360e19010440e946a4bb8f564610d P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:37.947927 10748 tablet.cc:1722] T b64360e19010440e946a4bb8f564610d P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:37.949044 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=568d7289694346268a4fc7e5d46a06ff] alter complete (version 3)
I20240503 13:59:38.050797 6383 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:38.050942 6383 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:38.053627 6383 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet b64360e19010440e946a4bb8f564610d
I20240503 13:59:38.054205 9058 tablet_service.cc:1514] Processing DeleteTablet for tablet b64360e19010440e946a4bb8f564610d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:38 UTC) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:38.054255 6769 tablet_service.cc:1514] Processing DeleteTablet for tablet b64360e19010440e946a4bb8f564610d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:38 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:38.054590 6902 tablet_service.cc:1514] Processing DeleteTablet for tablet b64360e19010440e946a4bb8f564610d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:38 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:38.054666 10758 tablet_replica.cc:330] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:38.054675 10757 tablet_replica.cc:330] T b64360e19010440e946a4bb8f564610d P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:38.054818 10758 raft_consensus.cc:2229] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:38.054821 10757 raft_consensus.cc:2229] T b64360e19010440e946a4bb8f564610d P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:38.054885 10759 tablet_replica.cc:330] T b64360e19010440e946a4bb8f564610d P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:38.054960 10757 raft_consensus.cc:2258] T b64360e19010440e946a4bb8f564610d P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:38.055068 10758 raft_consensus.cc:2258] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:38.055068 10759 raft_consensus.cc:2229] T b64360e19010440e946a4bb8f564610d P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:38.055186 10759 raft_consensus.cc:2258] T b64360e19010440e946a4bb8f564610d P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:38.055708 10757 ts_tablet_manager.cc:1875] T b64360e19010440e946a4bb8f564610d P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:38.055744 10759 ts_tablet_manager.cc:1875] T b64360e19010440e946a4bb8f564610d P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:38.057899 10759 ts_tablet_manager.cc:1888] T b64360e19010440e946a4bb8f564610d P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:38.057940 10757 ts_tablet_manager.cc:1888] T b64360e19010440e946a4bb8f564610d P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:38.057968 10759 log.cc:1196] T b64360e19010440e946a4bb8f564610d P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/b64360e19010440e946a4bb8f564610d
I20240503 13:59:38.057994 10757 log.cc:1196] T b64360e19010440e946a4bb8f564610d P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/b64360e19010440e946a4bb8f564610d
I20240503 13:59:38.058269 10759 ts_tablet_manager.cc:1909] T b64360e19010440e946a4bb8f564610d P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:38.058270 10757 ts_tablet_manager.cc:1909] T b64360e19010440e946a4bb8f564610d P 8d574fb4a89d4eea947c859470d44dd4: Deleting consensus metadata
I20240503 13:59:38.058703 10758 ts_tablet_manager.cc:1875] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:38.058928 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet b64360e19010440e946a4bb8f564610d (table rebalancer_dynamic_table_new_name [id=568d7289694346268a4fc7e5d46a06ff]) successfully deleted
I20240503 13:59:38.059091 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet b64360e19010440e946a4bb8f564610d (table rebalancer_dynamic_table_new_name [id=568d7289694346268a4fc7e5d46a06ff]) successfully deleted
I20240503 13:59:38.060514 10758 ts_tablet_manager.cc:1888] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:38.060575 10758 log.cc:1196] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/b64360e19010440e946a4bb8f564610d
I20240503 13:59:38.060838 10758 ts_tablet_manager.cc:1909] T b64360e19010440e946a4bb8f564610d P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:38.061393 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet b64360e19010440e946a4bb8f564610d (table rebalancer_dynamic_table_new_name [id=568d7289694346268a4fc7e5d46a06ff]) successfully deleted
I20240503 13:59:38.156030 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:38.161550 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 01005de9b6f4467bbcdab96d0b5901d7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=003c8a6d488f4400b476e28f0eeaf2c5]), partition=
I20240503 13:59:38.161839 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01005de9b6f4467bbcdab96d0b5901d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:38.163542 6902 tablet_service.cc:1467] Processing CreateTablet for tablet 01005de9b6f4467bbcdab96d0b5901d7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=003c8a6d488f4400b476e28f0eeaf2c5]), partition=
I20240503 13:59:38.163769 6902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01005de9b6f4467bbcdab96d0b5901d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:38.163889 10727 tablet_bootstrap.cc:492] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:38.164877 10727 tablet_bootstrap.cc:654] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:38.165848 10710 tablet_bootstrap.cc:492] T 01005de9b6f4467bbcdab96d0b5901d7 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:38.165959 10727 tablet_bootstrap.cc:492] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:38.166054 10727 ts_tablet_manager.cc:1367] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:38.166244 9191 tablet_service.cc:1467] Processing CreateTablet for tablet 01005de9b6f4467bbcdab96d0b5901d7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=003c8a6d488f4400b476e28f0eeaf2c5]), partition=
I20240503 13:59:38.166486 9191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01005de9b6f4467bbcdab96d0b5901d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:38.166524 10727 raft_consensus.cc:348] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:38.166641 10727 raft_consensus.cc:374] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:38.166667 10710 tablet_bootstrap.cc:654] T 01005de9b6f4467bbcdab96d0b5901d7 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:38.166689 10727 raft_consensus.cc:729] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:38.166811 10727 consensus_queue.cc:260] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:38.167083 10727 ts_tablet_manager.cc:1398] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:38.167954 10710 tablet_bootstrap.cc:492] T 01005de9b6f4467bbcdab96d0b5901d7 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:38.168045 10710 ts_tablet_manager.cc:1367] T 01005de9b6f4467bbcdab96d0b5901d7 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:38.168401 10710 raft_consensus.cc:348] T 01005de9b6f4467bbcdab96d0b5901d7 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:38.168496 10710 raft_consensus.cc:374] T 01005de9b6f4467bbcdab96d0b5901d7 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:38.168531 10710 raft_consensus.cc:729] T 01005de9b6f4467bbcdab96d0b5901d7 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:38.168684 10762 tablet_bootstrap.cc:492] T 01005de9b6f4467bbcdab96d0b5901d7 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:38.168659 10710 consensus_queue.cc:260] T 01005de9b6f4467bbcdab96d0b5901d7 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:38.168895 10710 ts_tablet_manager.cc:1398] T 01005de9b6f4467bbcdab96d0b5901d7 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:38.169585 10762 tablet_bootstrap.cc:654] T 01005de9b6f4467bbcdab96d0b5901d7 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:38.170811 10762 tablet_bootstrap.cc:492] T 01005de9b6f4467bbcdab96d0b5901d7 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:38.170943 10762 ts_tablet_manager.cc:1367] T 01005de9b6f4467bbcdab96d0b5901d7 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:38.171483 10762 raft_consensus.cc:348] T 01005de9b6f4467bbcdab96d0b5901d7 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:38.171700 10762 raft_consensus.cc:374] T 01005de9b6f4467bbcdab96d0b5901d7 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:38.171777 10762 raft_consensus.cc:729] T 01005de9b6f4467bbcdab96d0b5901d7 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:38.171919 10762 consensus_queue.cc:260] T 01005de9b6f4467bbcdab96d0b5901d7 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:38.172384 10762 ts_tablet_manager.cc:1398] T 01005de9b6f4467bbcdab96d0b5901d7 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:38.188998 10675 consensus_queue.cc:1035] T 82f1b400e34743db819a831b73365e00 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:38.193271 10716 consensus_queue.cc:1035] T 82f1b400e34743db819a831b73365e00 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:38.243232 10751 consensus_queue.cc:1035] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:38.247999 10751 consensus_queue.cc:1035] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:38.307950 10751 consensus_queue.cc:1035] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:38.311944 10564 consensus_queue.cc:1035] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:38.375296 6386 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:38.381556 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 133942ceb914456da3f1a281f16d578f (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=5acb9e8cbce6439bbadafaf4632200c2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:38.381892 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133942ceb914456da3f1a281f16d578f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:38.381944 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 9a6709e4024d47448c20d33a2b36479e (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=5acb9e8cbce6439bbadafaf4632200c2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:38.382225 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a6709e4024d47448c20d33a2b36479e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:38.382328 6769 tablet_service.cc:1467] Processing CreateTablet for tablet 133942ceb914456da3f1a281f16d578f (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=5acb9e8cbce6439bbadafaf4632200c2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:38.382378 6771 tablet_service.cc:1467] Processing CreateTablet for tablet 37c1f03606fe4e5f8e31a252cce18016 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=5acb9e8cbce6439bbadafaf4632200c2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:38.382570 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133942ceb914456da3f1a281f16d578f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:38.382689 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c1f03606fe4e5f8e31a252cce18016. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:38.384444 9191 tablet_service.cc:1467] Processing CreateTablet for tablet 133942ceb914456da3f1a281f16d578f (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=5acb9e8cbce6439bbadafaf4632200c2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:38.384737 9191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133942ceb914456da3f1a281f16d578f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:38.385082 10769 raft_consensus.cc:482] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:38.385164 10769 raft_consensus.cc:504] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:38.385257 9192 tablet_service.cc:1467] Processing CreateTablet for tablet 37c1f03606fe4e5f8e31a252cce18016 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=5acb9e8cbce6439bbadafaf4632200c2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:38.385526 9192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c1f03606fe4e5f8e31a252cce18016. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:38.385598 10769 leader_election.cc:288] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:38.386787 10762 tablet_bootstrap.cc:492] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:38.386955 9190 tablet_service.cc:1467] Processing CreateTablet for tablet 9a6709e4024d47448c20d33a2b36479e (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=5acb9e8cbce6439bbadafaf4632200c2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:38.387130 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a6709e4024d47448c20d33a2b36479e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:38.387827 10762 tablet_bootstrap.cc:654] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:38.388084 10776 tablet_bootstrap.cc:492] T 133942ceb914456da3f1a281f16d578f P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:38.389194 10727 tablet_bootstrap.cc:492] T 133942ceb914456da3f1a281f16d578f P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:38.389621 10776 tablet_bootstrap.cc:654] T 133942ceb914456da3f1a281f16d578f P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:38.389686 10762 tablet_bootstrap.cc:492] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:38.389765 10762 ts_tablet_manager.cc:1367] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:59:38.390064 10727 tablet_bootstrap.cc:654] T 133942ceb914456da3f1a281f16d578f P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:38.390283 10762 raft_consensus.cc:348] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:38.390379 10762 raft_consensus.cc:374] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:38.390404 10762 raft_consensus.cc:729] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:38.390513 10762 consensus_queue.cc:260] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:38.390710 10776 tablet_bootstrap.cc:492] T 133942ceb914456da3f1a281f16d578f P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:38.390786 10776 ts_tablet_manager.cc:1367] T 133942ceb914456da3f1a281f16d578f P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:38.390836 10762 ts_tablet_manager.cc:1398] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:38.390842 6902 tablet_service.cc:1467] Processing CreateTablet for tablet 37c1f03606fe4e5f8e31a252cce18016 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=5acb9e8cbce6439bbadafaf4632200c2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:38.391014 10762 tablet_bootstrap.cc:492] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:38.391108 6902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c1f03606fe4e5f8e31a252cce18016. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:38.391109 10727 tablet_bootstrap.cc:492] T 133942ceb914456da3f1a281f16d578f P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:38.391184 10727 ts_tablet_manager.cc:1367] T 133942ceb914456da3f1a281f16d578f P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:38.391247 10776 raft_consensus.cc:348] T 133942ceb914456da3f1a281f16d578f P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:38.391366 10776 raft_consensus.cc:374] T 133942ceb914456da3f1a281f16d578f P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:38.391408 10776 raft_consensus.cc:729] T 133942ceb914456da3f1a281f16d578f P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:38.391527 10776 consensus_queue.cc:260] T 133942ceb914456da3f1a281f16d578f P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:38.391647 10727 raft_consensus.cc:348] T 133942ceb914456da3f1a281f16d578f P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:38.391741 10727 raft_consensus.cc:374] T 133942ceb914456da3f1a281f16d578f P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:38.391768 10727 raft_consensus.cc:729] T 133942ceb914456da3f1a281f16d578f P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:38.391777 10776 ts_tablet_manager.cc:1398] T 133942ceb914456da3f1a281f16d578f P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:38.391863 10727 consensus_queue.cc:260] T 133942ceb914456da3f1a281f16d578f P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:38.391927 10776 tablet_bootstrap.cc:492] T 37c1f03606fe4e5f8e31a252cce18016 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:38.392129 10727 ts_tablet_manager.cc:1398] T 133942ceb914456da3f1a281f16d578f P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:38.392827 10762 tablet_bootstrap.cc:654] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:38.393086 10776 tablet_bootstrap.cc:654] T 37c1f03606fe4e5f8e31a252cce18016 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:38.393311 10710 tablet_bootstrap.cc:492] T 37c1f03606fe4e5f8e31a252cce18016 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:38.394258 10710 tablet_bootstrap.cc:654] T 37c1f03606fe4e5f8e31a252cce18016 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:38.394469 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 9a6709e4024d47448c20d33a2b36479e (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=5acb9e8cbce6439bbadafaf4632200c2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:38.394734 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a6709e4024d47448c20d33a2b36479e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:38.395677 10776 tablet_bootstrap.cc:492] T 37c1f03606fe4e5f8e31a252cce18016 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:38.395764 10776 ts_tablet_manager.cc:1367] T 37c1f03606fe4e5f8e31a252cce18016 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:38.395942 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01005de9b6f4467bbcdab96d0b5901d7" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:38.396077 9211 raft_consensus.cc:2454] T 01005de9b6f4467bbcdab96d0b5901d7 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:38.396196 10776 raft_consensus.cc:348] T 37c1f03606fe4e5f8e31a252cce18016 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:38.396286 10776 raft_consensus.cc:374] T 37c1f03606fe4e5f8e31a252cce18016 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:38.396314 10776 raft_consensus.cc:729] T 37c1f03606fe4e5f8e31a252cce18016 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:38.396418 10776 consensus_queue.cc:260] T 37c1f03606fe4e5f8e31a252cce18016 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:38.396709 10776 ts_tablet_manager.cc:1398] T 37c1f03606fe4e5f8e31a252cce18016 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:38.397287 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01005de9b6f4467bbcdab96d0b5901d7" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:38.397400 6922 raft_consensus.cc:2454] T 01005de9b6f4467bbcdab96d0b5901d7 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:38.397583 8883 leader_election.cc:302] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:38.397845 10769 raft_consensus.cc:2781] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:38.397898 10769 raft_consensus.cc:482] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:38.397934 10769 raft_consensus.cc:3037] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:38.398772 10783 tablet_bootstrap.cc:492] T 9a6709e4024d47448c20d33a2b36479e P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:38.399060 10769 raft_consensus.cc:504] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:38.399461 10769 leader_election.cc:288] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:38.399807 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01005de9b6f4467bbcdab96d0b5901d7" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:38.399909 6922 raft_consensus.cc:3037] T 01005de9b6f4467bbcdab96d0b5901d7 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:38.400004 10710 tablet_bootstrap.cc:492] T 37c1f03606fe4e5f8e31a252cce18016 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:38.400110 10710 ts_tablet_manager.cc:1367] T 37c1f03606fe4e5f8e31a252cce18016 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240503 13:59:38.400099 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01005de9b6f4467bbcdab96d0b5901d7" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:38.400207 9211 raft_consensus.cc:3037] T 01005de9b6f4467bbcdab96d0b5901d7 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:38.400542 10710 raft_consensus.cc:348] T 37c1f03606fe4e5f8e31a252cce18016 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:38.400635 10710 raft_consensus.cc:374] T 37c1f03606fe4e5f8e31a252cce18016 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:38.400666 10710 raft_consensus.cc:729] T 37c1f03606fe4e5f8e31a252cce18016 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:38.400753 10783 tablet_bootstrap.cc:654] T 9a6709e4024d47448c20d33a2b36479e P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:38.400782 10710 consensus_queue.cc:260] T 37c1f03606fe4e5f8e31a252cce18016 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:38.400902 6922 raft_consensus.cc:2454] T 01005de9b6f4467bbcdab96d0b5901d7 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:38.401109 10710 ts_tablet_manager.cc:1398] T 37c1f03606fe4e5f8e31a252cce18016 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:38.401232 9211 raft_consensus.cc:2454] T 01005de9b6f4467bbcdab96d0b5901d7 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:38.401332 8883 leader_election.cc:302] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:38.401742 10731 raft_consensus.cc:2781] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:38.401816 10731 raft_consensus.cc:686] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 13:59:38.401947 10731 consensus_queue.cc:237] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:38.405701 10775 tablet_bootstrap.cc:492] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:38.406351 10762 tablet_bootstrap.cc:492] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:38.406494 10762 ts_tablet_manager.cc:1367] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 13:59:38.406877 6383 catalog_manager.cc:5484] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:38.407039 10762 raft_consensus.cc:348] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:38.407142 10762 raft_consensus.cc:374] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:38.407174 10762 raft_consensus.cc:729] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:38.407320 10762 consensus_queue.cc:260] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:38.408309 10762 ts_tablet_manager.cc:1398] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:38.408567 10762 tablet_bootstrap.cc:492] T 9a6709e4024d47448c20d33a2b36479e P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:38.409509 10783 tablet_bootstrap.cc:492] T 9a6709e4024d47448c20d33a2b36479e P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:38.409621 10783 ts_tablet_manager.cc:1367] T 9a6709e4024d47448c20d33a2b36479e P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20240503 13:59:38.410339 10783 raft_consensus.cc:348] T 9a6709e4024d47448c20d33a2b36479e P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:38.410501 10783 raft_consensus.cc:374] T 9a6709e4024d47448c20d33a2b36479e P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:38.410578 10783 raft_consensus.cc:729] T 9a6709e4024d47448c20d33a2b36479e P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:38.410768 10783 consensus_queue.cc:260] T 9a6709e4024d47448c20d33a2b36479e P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:38.411129 10783 ts_tablet_manager.cc:1398] T 9a6709e4024d47448c20d33a2b36479e P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:38.410796 10762 tablet_bootstrap.cc:654] T 9a6709e4024d47448c20d33a2b36479e P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:38.413017 10762 tablet_bootstrap.cc:492] T 9a6709e4024d47448c20d33a2b36479e P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:38.413139 10762 ts_tablet_manager.cc:1367] T 9a6709e4024d47448c20d33a2b36479e P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:38.413259 10775 tablet_bootstrap.cc:654] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:38.413730 10762 raft_consensus.cc:348] T 9a6709e4024d47448c20d33a2b36479e P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:38.413843 10762 raft_consensus.cc:374] T 9a6709e4024d47448c20d33a2b36479e P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:38.413899 10762 raft_consensus.cc:729] T 9a6709e4024d47448c20d33a2b36479e P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:38.414103 10762 consensus_queue.cc:260] T 9a6709e4024d47448c20d33a2b36479e P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:38.414397 10762 ts_tablet_manager.cc:1398] T 9a6709e4024d47448c20d33a2b36479e P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:38.414666 10775 tablet_bootstrap.cc:492] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:38.414779 10775 ts_tablet_manager.cc:1367] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:59:38.415670 10775 raft_consensus.cc:348] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:38.415786 10775 raft_consensus.cc:374] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:38.415866 10775 raft_consensus.cc:729] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:38.416015 10775 consensus_queue.cc:260] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:38.416294 10775 ts_tablet_manager.cc:1398] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240503 13:59:38.462608 6838 tablet.cc:2369] T 133942ceb914456da3f1a281f16d578f P 4a85e492c8154e9881a9cc2a105d7e64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:59:38.503122 10724 raft_consensus.cc:482] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:38.503233 10724 raft_consensus.cc:504] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:38.503723 10724 leader_election.cc:288] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:38.503998 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6709e4024d47448c20d33a2b36479e" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:38.504047 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6709e4024d47448c20d33a2b36479e" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:38.504129 6660 raft_consensus.cc:2454] T 9a6709e4024d47448c20d33a2b36479e P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:38.504200 9211 raft_consensus.cc:2454] T 9a6709e4024d47448c20d33a2b36479e P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:38.504423 6465 leader_election.cc:302] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:38.504626 10724 raft_consensus.cc:2781] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:38.504684 10724 raft_consensus.cc:482] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:38.504712 10724 raft_consensus.cc:3037] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:38.505580 10724 raft_consensus.cc:504] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:38.505893 10724 leader_election.cc:288] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Requested vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:38.506158 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6709e4024d47448c20d33a2b36479e" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:38.506168 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6709e4024d47448c20d33a2b36479e" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:38.506256 9211 raft_consensus.cc:3037] T 9a6709e4024d47448c20d33a2b36479e P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:38.506260 6660 raft_consensus.cc:3037] T 9a6709e4024d47448c20d33a2b36479e P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:38.506987 9211 raft_consensus.cc:2454] T 9a6709e4024d47448c20d33a2b36479e P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:38.507023 6660 raft_consensus.cc:2454] T 9a6709e4024d47448c20d33a2b36479e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:38.507253 6464 leader_election.cc:302] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:38.507419 10724 raft_consensus.cc:2781] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:38.507592 10724 raft_consensus.cc:686] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Becoming Leader. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Running, Role: LEADER
I20240503 13:59:38.507745 10724 consensus_queue.cc:237] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:38.509841 6379 catalog_manager.cc:5484] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:38.528925 10784 raft_consensus.cc:482] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:38.529053 10784 raft_consensus.cc:504] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:38.529582 10784 leader_election.cc:288] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:38.530009 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133942ceb914456da3f1a281f16d578f" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:38.530035 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133942ceb914456da3f1a281f16d578f" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:38.530151 8947 raft_consensus.cc:2454] T 133942ceb914456da3f1a281f16d578f P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:38.530176 6791 raft_consensus.cc:2454] T 133942ceb914456da3f1a281f16d578f P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:38.530547 9146 leader_election.cc:302] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:38.530979 10784 raft_consensus.cc:2781] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:38.531047 10784 raft_consensus.cc:482] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:38.531090 10784 raft_consensus.cc:3037] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:38.532251 10784 raft_consensus.cc:504] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:38.532765 10784 leader_election.cc:288] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Requested vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:38.532881 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133942ceb914456da3f1a281f16d578f" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:38.533043 8948 raft_consensus.cc:3037] T 133942ceb914456da3f1a281f16d578f P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:38.533766 8948 raft_consensus.cc:2454] T 133942ceb914456da3f1a281f16d578f P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:38.534022 9147 leader_election.cc:302] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:38.534178 10784 raft_consensus.cc:2781] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:38.534260 10784 raft_consensus.cc:686] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 13:59:38.534415 10784 consensus_queue.cc:237] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:38.535346 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133942ceb914456da3f1a281f16d578f" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:38.535459 6791 raft_consensus.cc:3037] T 133942ceb914456da3f1a281f16d578f P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:38.536238 6791 raft_consensus.cc:2454] T 133942ceb914456da3f1a281f16d578f P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:38.536545 6379 catalog_manager.cc:5484] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:38.562222 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:38.565829 9205 raft_consensus.cc:1261] T 01005de9b6f4467bbcdab96d0b5901d7 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:38.565907 6920 raft_consensus.cc:1261] T 01005de9b6f4467bbcdab96d0b5901d7 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:38.566377 10731 consensus_queue.cc:1035] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:38.566675 10731 consensus_queue.cc:1035] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:38.571676 10768 mvcc.cc:205] Tried to move back new op lower bound from 7023594613001863168 to 7023594612335484928. Current Snapshot: MvccSnapshot[applied={T|T < 7023594613001863168}]
I20240503 13:59:38.572811 6380 catalog_manager.cc:5576] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091) does not have the latest schema for tablet 01005de9b6f4467bbcdab96d0b5901d7 (table rebalancer_dynamic_table [id=003c8a6d488f4400b476e28f0eeaf2c5]). Expected version 1 got 0
I20240503 13:59:38.574362 10792 mvcc.cc:205] Tried to move back new op lower bound from 7023594613001863168 to 7023594612335484928. Current Snapshot: MvccSnapshot[applied={T|T < 7023594613001863168}]
I20240503 13:59:38.574662 6383 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet 01005de9b6f4467bbcdab96d0b5901d7 (table rebalancer_dynamic_table [id=003c8a6d488f4400b476e28f0eeaf2c5]). Expected version 1 got 0
I20240503 13:59:38.575258 10797 tablet.cc:1722] T 01005de9b6f4467bbcdab96d0b5901d7 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:38.575763 10797 tablet.cc:1620] T 01005de9b6f4467bbcdab96d0b5901d7 P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:38.576314 6383 catalog_manager.cc:5576] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091) does not have the latest schema for tablet 01005de9b6f4467bbcdab96d0b5901d7 (table rebalancer_dynamic_table [id=003c8a6d488f4400b476e28f0eeaf2c5]). Expected version 1 got 0
I20240503 13:59:38.577347 6380 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet 01005de9b6f4467bbcdab96d0b5901d7 (table rebalancer_dynamic_table [id=003c8a6d488f4400b476e28f0eeaf2c5]). Expected version 1 got 0
I20240503 13:59:38.577595 10795 tablet.cc:1722] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:38.577929 10795 tablet.cc:1620] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:38.578649 6379 catalog_manager.cc:5576] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531) does not have the latest schema for tablet 01005de9b6f4467bbcdab96d0b5901d7 (table rebalancer_dynamic_table [id=003c8a6d488f4400b476e28f0eeaf2c5]). Expected version 1 got 0
I20240503 13:59:38.579725 10798 tablet.cc:1722] T 01005de9b6f4467bbcdab96d0b5901d7 P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:38.580025 10798 tablet.cc:1620] T 01005de9b6f4467bbcdab96d0b5901d7 P f58065db70b64ef8b1dc499aa6ae4460: MemRowSet was empty: no flush needed.
I20240503 13:59:38.583879 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=003c8a6d488f4400b476e28f0eeaf2c5] alter complete (version 1)
I20240503 13:59:38.654958 10796 raft_consensus.cc:482] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:38.655088 10796 raft_consensus.cc:504] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:38.656096 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c1f03606fe4e5f8e31a252cce18016" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:38.656247 6790 raft_consensus.cc:2454] T 37c1f03606fe4e5f8e31a252cce18016 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:38.656477 10796 leader_election.cc:288] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:38.656584 9146 leader_election.cc:302] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:38.656800 10796 raft_consensus.cc:2781] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:38.656859 10796 raft_consensus.cc:482] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:38.656909 10796 raft_consensus.cc:3037] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:38.657932 10796 raft_consensus.cc:504] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:38.658401 10796 leader_election.cc:288] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:38.658602 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c1f03606fe4e5f8e31a252cce18016" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:38.658726 6790 raft_consensus.cc:3037] T 37c1f03606fe4e5f8e31a252cce18016 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:38.659761 6790 raft_consensus.cc:2454] T 37c1f03606fe4e5f8e31a252cce18016 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:38.660020 9146 leader_election.cc:302] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:38.660187 10796 raft_consensus.cc:2781] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:38.660331 10796 raft_consensus.cc:686] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 13:59:38.660290 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c1f03606fe4e5f8e31a252cce18016" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:38.660290 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c1f03606fe4e5f8e31a252cce18016" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:38.660445 6920 raft_consensus.cc:2454] T 37c1f03606fe4e5f8e31a252cce18016 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:38.660969 10796 consensus_queue.cc:237] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 13:59:38.663175 6379 catalog_manager.cc:5484] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:38.686282 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:38.691355 10795 tablet.cc:1722] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:38.691718 10795 tablet.cc:1620] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:38.691802 10797 tablet.cc:1722] T 01005de9b6f4467bbcdab96d0b5901d7 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:38.692067 10797 tablet.cc:1620] T 01005de9b6f4467bbcdab96d0b5901d7 P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:38.692934 10798 tablet.cc:1722] T 01005de9b6f4467bbcdab96d0b5901d7 P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:38.693245 10798 tablet.cc:1620] T 01005de9b6f4467bbcdab96d0b5901d7 P f58065db70b64ef8b1dc499aa6ae4460: MemRowSet was empty: no flush needed.
I20240503 13:59:38.697175 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=003c8a6d488f4400b476e28f0eeaf2c5] alter complete (version 2)
I20240503 13:59:38.798864 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:38.805202 10795 tablet.cc:1722] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:38.805786 10798 tablet.cc:1722] T 01005de9b6f4467bbcdab96d0b5901d7 P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:38.805902 10797 tablet.cc:1722] T 01005de9b6f4467bbcdab96d0b5901d7 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:38.810526 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=003c8a6d488f4400b476e28f0eeaf2c5] alter complete (version 3)
I20240503 13:59:38.902032 10724 consensus_queue.cc:1035] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:38.907912 10724 consensus_queue.cc:1035] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:38.912405 6386 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:38.912549 6386 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:38.914445 6386 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 01005de9b6f4467bbcdab96d0b5901d7
I20240503 13:59:38.915199 8928 tablet_service.cc:1514] Processing DeleteTablet for tablet 01005de9b6f4467bbcdab96d0b5901d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:38 UTC) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:38.915199 9190 tablet_service.cc:1514] Processing DeleteTablet for tablet 01005de9b6f4467bbcdab96d0b5901d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:38 UTC) from {username='slave'} at 127.0.0.1:57960
I20240503 13:59:38.915489 6902 tablet_service.cc:1514] Processing DeleteTablet for tablet 01005de9b6f4467bbcdab96d0b5901d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:38 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:38.915513 10810 tablet_replica.cc:330] T 01005de9b6f4467bbcdab96d0b5901d7 P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 13:59:38.915668 10810 raft_consensus.cc:2229] T 01005de9b6f4467bbcdab96d0b5901d7 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:38.915791 10811 tablet_replica.cc:330] T 01005de9b6f4467bbcdab96d0b5901d7 P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:38.915791 10810 raft_consensus.cc:2258] T 01005de9b6f4467bbcdab96d0b5901d7 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:38.915910 10811 raft_consensus.cc:2229] T 01005de9b6f4467bbcdab96d0b5901d7 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:38.916018 10809 tablet_replica.cc:330] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:38.916028 10811 raft_consensus.cc:2258] T 01005de9b6f4467bbcdab96d0b5901d7 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:38.916173 10809 raft_consensus.cc:2229] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:38.916520 10810 ts_tablet_manager.cc:1875] T 01005de9b6f4467bbcdab96d0b5901d7 P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:38.916540 10811 ts_tablet_manager.cc:1875] T 01005de9b6f4467bbcdab96d0b5901d7 P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:38.916527 10809 raft_consensus.cc:2258] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:38.917130 10809 ts_tablet_manager.cc:1875] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:38.919219 10811 ts_tablet_manager.cc:1888] T 01005de9b6f4467bbcdab96d0b5901d7 P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:38.919346 10811 log.cc:1196] T 01005de9b6f4467bbcdab96d0b5901d7 P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/01005de9b6f4467bbcdab96d0b5901d7
I20240503 13:59:38.919544 10810 ts_tablet_manager.cc:1888] T 01005de9b6f4467bbcdab96d0b5901d7 P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:38.919651 10810 log.cc:1196] T 01005de9b6f4467bbcdab96d0b5901d7 P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/01005de9b6f4467bbcdab96d0b5901d7
I20240503 13:59:38.919714 10811 ts_tablet_manager.cc:1909] T 01005de9b6f4467bbcdab96d0b5901d7 P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:38.919770 10809 ts_tablet_manager.cc:1888] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:38.919863 10809 log.cc:1196] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/01005de9b6f4467bbcdab96d0b5901d7
I20240503 13:59:38.919926 10810 ts_tablet_manager.cc:1909] T 01005de9b6f4467bbcdab96d0b5901d7 P ab1d21935d6049888eb0ceccc3c9d735: Deleting consensus metadata
I20240503 13:59:38.920156 10809 ts_tablet_manager.cc:1909] T 01005de9b6f4467bbcdab96d0b5901d7 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting consensus metadata
I20240503 13:59:38.920398 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet 01005de9b6f4467bbcdab96d0b5901d7 (table rebalancer_dynamic_table_new_name [id=003c8a6d488f4400b476e28f0eeaf2c5]) successfully deleted
I20240503 13:59:38.920533 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet 01005de9b6f4467bbcdab96d0b5901d7 (table rebalancer_dynamic_table_new_name [id=003c8a6d488f4400b476e28f0eeaf2c5]) successfully deleted
I20240503 13:59:38.921150 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 01005de9b6f4467bbcdab96d0b5901d7 (table rebalancer_dynamic_table_new_name [id=003c8a6d488f4400b476e28f0eeaf2c5]) successfully deleted
I20240503 13:59:39.016184 6386 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:39.022006 9321 tablet_service.cc:1467] Processing CreateTablet for tablet 98e8b7edc671479baeabd247d4ae686b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9cc21533093646e3bf8f128856c19ccc]), partition=
I20240503 13:59:39.022276 9321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e8b7edc671479baeabd247d4ae686b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:39.022284 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 98e8b7edc671479baeabd247d4ae686b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9cc21533093646e3bf8f128856c19ccc]), partition=
I20240503 13:59:39.022493 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e8b7edc671479baeabd247d4ae686b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:39.024236 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 98e8b7edc671479baeabd247d4ae686b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9cc21533093646e3bf8f128856c19ccc]), partition=
I20240503 13:59:39.024482 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e8b7edc671479baeabd247d4ae686b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:39.025492 10812 tablet_bootstrap.cc:492] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:39.025557 10813 tablet_bootstrap.cc:492] T 98e8b7edc671479baeabd247d4ae686b P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:39.026327 10813 tablet_bootstrap.cc:654] T 98e8b7edc671479baeabd247d4ae686b P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:39.026681 10812 tablet_bootstrap.cc:654] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:39.026887 10814 tablet_bootstrap.cc:492] T 98e8b7edc671479baeabd247d4ae686b P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:39.027714 10814 tablet_bootstrap.cc:654] T 98e8b7edc671479baeabd247d4ae686b P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:39.028110 10812 tablet_bootstrap.cc:492] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:39.028232 10812 ts_tablet_manager.cc:1367] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:39.029093 10812 raft_consensus.cc:348] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:39.029217 10812 raft_consensus.cc:374] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:39.029217 10814 tablet_bootstrap.cc:492] T 98e8b7edc671479baeabd247d4ae686b P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:39.029250 10812 raft_consensus.cc:729] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:39.029295 10814 ts_tablet_manager.cc:1367] T 98e8b7edc671479baeabd247d4ae686b P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:39.029405 10812 consensus_queue.cc:260] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:39.029743 10814 raft_consensus.cc:348] T 98e8b7edc671479baeabd247d4ae686b P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:39.029839 10812 ts_tablet_manager.cc:1398] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:39.029889 10814 raft_consensus.cc:374] T 98e8b7edc671479baeabd247d4ae686b P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:39.029906 10813 tablet_bootstrap.cc:492] T 98e8b7edc671479baeabd247d4ae686b P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:39.029932 10814 raft_consensus.cc:729] T 98e8b7edc671479baeabd247d4ae686b P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:39.029983 10813 ts_tablet_manager.cc:1367] T 98e8b7edc671479baeabd247d4ae686b P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:39.030079 10814 consensus_queue.cc:260] T 98e8b7edc671479baeabd247d4ae686b P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:39.030354 10814 ts_tablet_manager.cc:1398] T 98e8b7edc671479baeabd247d4ae686b P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:39.030455 10813 raft_consensus.cc:348] T 98e8b7edc671479baeabd247d4ae686b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:39.030555 10813 raft_consensus.cc:374] T 98e8b7edc671479baeabd247d4ae686b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:39.030577 10813 raft_consensus.cc:729] T 98e8b7edc671479baeabd247d4ae686b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:39.030668 10813 consensus_queue.cc:260] T 98e8b7edc671479baeabd247d4ae686b P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:39.030858 10813 ts_tablet_manager.cc:1398] T 98e8b7edc671479baeabd247d4ae686b P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:39.076608 10784 consensus_queue.cc:1035] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:39.080356 10796 consensus_queue.cc:1035] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:39.086921 10796 consensus_queue.cc:1035] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:39.089706 6920 raft_consensus.cc:3037] T 37c1f03606fe4e5f8e31a252cce18016 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:39.091883 10796 consensus_queue.cc:1035] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:39.099877 10750 raft_consensus.cc:482] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:39.099973 10750 raft_consensus.cc:504] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:39.100394 10750 leader_election.cc:288] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:39.100746 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e8b7edc671479baeabd247d4ae686b" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:39.100878 6527 raft_consensus.cc:2454] T 98e8b7edc671479baeabd247d4ae686b P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:39.100827 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e8b7edc671479baeabd247d4ae686b" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:39.100980 8948 raft_consensus.cc:2454] T 98e8b7edc671479baeabd247d4ae686b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:39.101181 9278 leader_election.cc:302] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:39.101465 10750 raft_consensus.cc:2781] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:39.101516 10750 raft_consensus.cc:482] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:39.101560 10750 raft_consensus.cc:3037] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:39.102566 10750 raft_consensus.cc:504] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:39.102849 10750 leader_election.cc:288] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:39.103266 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e8b7edc671479baeabd247d4ae686b" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:39.103416 6527 raft_consensus.cc:3037] T 98e8b7edc671479baeabd247d4ae686b P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:39.103602 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e8b7edc671479baeabd247d4ae686b" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:39.103710 8948 raft_consensus.cc:3037] T 98e8b7edc671479baeabd247d4ae686b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:39.104360 6527 raft_consensus.cc:2454] T 98e8b7edc671479baeabd247d4ae686b P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:39.104570 8948 raft_consensus.cc:2454] T 98e8b7edc671479baeabd247d4ae686b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:39.104713 9278 leader_election.cc:302] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:39.105027 10750 raft_consensus.cc:2781] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:39.105125 10750 raft_consensus.cc:686] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:39.105253 10750 consensus_queue.cc:237] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:39.107542 6386 catalog_manager.cc:5484] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:39.180783 6386 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:39.187156 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 9ac9791abf2b4a60ab7a6e4e2d4b0de6 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=483a5fbd5c7a44c8bd3bb20336c11e18]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:39.187130 9321 tablet_service.cc:1467] Processing CreateTablet for tablet 9ac9791abf2b4a60ab7a6e4e2d4b0de6 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=483a5fbd5c7a44c8bd3bb20336c11e18]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:39.187453 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ac9791abf2b4a60ab7a6e4e2d4b0de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:39.187456 9321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ac9791abf2b4a60ab7a6e4e2d4b0de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:39.187450 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 98466061b550450bb9d0041427f6f19d (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=483a5fbd5c7a44c8bd3bb20336c11e18]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:39.187729 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98466061b550450bb9d0041427f6f19d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:39.189630 10812 tablet_bootstrap.cc:492] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:39.189656 10824 tablet_bootstrap.cc:492] T 98466061b550450bb9d0041427f6f19d P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:39.190042 10814 tablet_bootstrap.cc:492] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:39.190431 9058 tablet_service.cc:1467] Processing CreateTablet for tablet e76ded33d780455d963c51a253cbd0e0 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=483a5fbd5c7a44c8bd3bb20336c11e18]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:39.190605 10812 tablet_bootstrap.cc:654] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:39.190639 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e76ded33d780455d963c51a253cbd0e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:39.190681 10824 tablet_bootstrap.cc:654] T 98466061b550450bb9d0041427f6f19d P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:39.191061 10814 tablet_bootstrap.cc:654] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:39.191799 6771 tablet_service.cc:1467] Processing CreateTablet for tablet 9ac9791abf2b4a60ab7a6e4e2d4b0de6 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=483a5fbd5c7a44c8bd3bb20336c11e18]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:39.192104 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ac9791abf2b4a60ab7a6e4e2d4b0de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:39.194772 10814 tablet_bootstrap.cc:492] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:39.194869 10814 ts_tablet_manager.cc:1367] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:39.195207 6640 tablet_service.cc:1467] Processing CreateTablet for tablet e76ded33d780455d963c51a253cbd0e0 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=483a5fbd5c7a44c8bd3bb20336c11e18]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:39.195478 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e76ded33d780455d963c51a253cbd0e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:39.195405 10814 raft_consensus.cc:348] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:39.195567 10814 raft_consensus.cc:374] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:39.195621 10814 raft_consensus.cc:729] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:39.195780 10814 consensus_queue.cc:260] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:39.196439 10812 tablet_bootstrap.cc:492] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:39.196524 10812 ts_tablet_manager.cc:1367] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:59:39.196542 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 98466061b550450bb9d0041427f6f19d (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=483a5fbd5c7a44c8bd3bb20336c11e18]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:39.196789 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98466061b550450bb9d0041427f6f19d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:39.196985 10812 raft_consensus.cc:348] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:39.197114 10812 raft_consensus.cc:374] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:39.197154 10812 raft_consensus.cc:729] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:39.197279 10812 consensus_queue.cc:260] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:39.197526 10812 ts_tablet_manager.cc:1398] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:39.197710 10829 tablet_bootstrap.cc:492] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:39.198428 9321 tablet_service.cc:1467] Processing CreateTablet for tablet 98466061b550450bb9d0041427f6f19d (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=483a5fbd5c7a44c8bd3bb20336c11e18]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:39.198596 10828 tablet_bootstrap.cc:492] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:39.198772 10829 tablet_bootstrap.cc:654] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:39.199194 9321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98466061b550450bb9d0041427f6f19d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:39.199615 10813 tablet_bootstrap.cc:492] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:39.199903 10828 tablet_bootstrap.cc:654] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:39.200644 10813 tablet_bootstrap.cc:654] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:39.200997 10828 tablet_bootstrap.cc:492] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:39.201102 10828 ts_tablet_manager.cc:1367] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:39.201100 10824 tablet_bootstrap.cc:492] T 98466061b550450bb9d0041427f6f19d P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:39.201189 10824 ts_tablet_manager.cc:1367] T 98466061b550450bb9d0041427f6f19d P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 13:59:39.201521 10828 raft_consensus.cc:348] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:39.201628 10828 raft_consensus.cc:374] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:39.201661 10828 raft_consensus.cc:729] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:39.201717 10824 raft_consensus.cc:348] T 98466061b550450bb9d0041427f6f19d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:39.201822 10824 raft_consensus.cc:374] T 98466061b550450bb9d0041427f6f19d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:39.201786 10828 consensus_queue.cc:260] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:39.201853 10824 raft_consensus.cc:729] T 98466061b550450bb9d0041427f6f19d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:39.201958 10824 consensus_queue.cc:260] T 98466061b550450bb9d0041427f6f19d P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:39.202073 10828 ts_tablet_manager.cc:1398] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:39.202220 10824 ts_tablet_manager.cc:1398] T 98466061b550450bb9d0041427f6f19d P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:39.202376 10824 tablet_bootstrap.cc:492] T e76ded33d780455d963c51a253cbd0e0 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:39.203706 10812 tablet_bootstrap.cc:492] T 98466061b550450bb9d0041427f6f19d P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:39.199213 9322 tablet_service.cc:1467] Processing CreateTablet for tablet e76ded33d780455d963c51a253cbd0e0 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=483a5fbd5c7a44c8bd3bb20336c11e18]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:39.204439 10829 tablet_bootstrap.cc:492] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:39.204517 10829 ts_tablet_manager.cc:1367] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:59:39.204591 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e76ded33d780455d963c51a253cbd0e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:39.204689 10812 tablet_bootstrap.cc:654] T 98466061b550450bb9d0041427f6f19d P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:39.204978 10829 raft_consensus.cc:348] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:39.205116 10829 raft_consensus.cc:374] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:39.205160 10829 raft_consensus.cc:729] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:39.205277 10829 consensus_queue.cc:260] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:39.205523 10829 ts_tablet_manager.cc:1398] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:39.207876 10813 tablet_bootstrap.cc:492] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:39.207979 10813 ts_tablet_manager.cc:1367] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:59:39.208487 10813 raft_consensus.cc:348] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:39.208595 10813 raft_consensus.cc:374] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:39.208628 10813 raft_consensus.cc:729] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:39.208747 10813 consensus_queue.cc:260] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:39.209022 10813 ts_tablet_manager.cc:1398] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:39.209314 10814 ts_tablet_manager.cc:1398] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240503 13:59:39.210542 10812 tablet_bootstrap.cc:492] T 98466061b550450bb9d0041427f6f19d P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:39.210625 10812 ts_tablet_manager.cc:1367] T 98466061b550450bb9d0041427f6f19d P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:59:39.211050 10812 raft_consensus.cc:348] T 98466061b550450bb9d0041427f6f19d P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:39.211149 10812 raft_consensus.cc:374] T 98466061b550450bb9d0041427f6f19d P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:39.211192 10812 raft_consensus.cc:729] T 98466061b550450bb9d0041427f6f19d P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:39.211347 10812 consensus_queue.cc:260] T 98466061b550450bb9d0041427f6f19d P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:39.211687 10812 ts_tablet_manager.cc:1398] T 98466061b550450bb9d0041427f6f19d P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:39.211854 10812 tablet_bootstrap.cc:492] T e76ded33d780455d963c51a253cbd0e0 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:39.212863 10812 tablet_bootstrap.cc:654] T e76ded33d780455d963c51a253cbd0e0 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:39.214844 10824 tablet_bootstrap.cc:654] T e76ded33d780455d963c51a253cbd0e0 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:39.216189 10812 tablet_bootstrap.cc:492] T e76ded33d780455d963c51a253cbd0e0 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:39.216284 10812 ts_tablet_manager.cc:1367] T e76ded33d780455d963c51a253cbd0e0 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 13:59:39.216404 10824 tablet_bootstrap.cc:492] T e76ded33d780455d963c51a253cbd0e0 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:39.216493 10824 ts_tablet_manager.cc:1367] T e76ded33d780455d963c51a253cbd0e0 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.003s
I20240503 13:59:39.216661 10812 raft_consensus.cc:348] T e76ded33d780455d963c51a253cbd0e0 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:39.216734 10812 raft_consensus.cc:374] T e76ded33d780455d963c51a253cbd0e0 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:39.216754 10812 raft_consensus.cc:729] T e76ded33d780455d963c51a253cbd0e0 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:39.216825 10812 consensus_queue.cc:260] T e76ded33d780455d963c51a253cbd0e0 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:39.216832 10824 raft_consensus.cc:348] T e76ded33d780455d963c51a253cbd0e0 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:39.216955 10824 raft_consensus.cc:374] T e76ded33d780455d963c51a253cbd0e0 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:39.216989 10824 raft_consensus.cc:729] T e76ded33d780455d963c51a253cbd0e0 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:39.217095 10812 ts_tablet_manager.cc:1398] T e76ded33d780455d963c51a253cbd0e0 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:39.217099 10824 consensus_queue.cc:260] T e76ded33d780455d963c51a253cbd0e0 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:39.217350 10824 ts_tablet_manager.cc:1398] T e76ded33d780455d963c51a253cbd0e0 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:39.227035 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:39.231487 8948 raft_consensus.cc:1261] T 98e8b7edc671479baeabd247d4ae686b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:39.231906 6527 raft_consensus.cc:1261] T 98e8b7edc671479baeabd247d4ae686b P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:39.231952 10751 consensus_queue.cc:1035] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:39.232255 10751 consensus_queue.cc:1035] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:39.234658 6386 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet 98e8b7edc671479baeabd247d4ae686b (table rebalancer_dynamic_table [id=9cc21533093646e3bf8f128856c19ccc]). Expected version 1 got 0
I20240503 13:59:39.234930 10795 tablet.cc:1722] T 98e8b7edc671479baeabd247d4ae686b P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:39.235231 10795 tablet.cc:1620] T 98e8b7edc671479baeabd247d4ae686b P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:39.236661 6386 catalog_manager.cc:5576] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931) does not have the latest schema for tablet 98e8b7edc671479baeabd247d4ae686b (table rebalancer_dynamic_table [id=9cc21533093646e3bf8f128856c19ccc]). Expected version 1 got 0
I20240503 13:59:39.237164 6379 catalog_manager.cc:5576] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021) does not have the latest schema for tablet 98e8b7edc671479baeabd247d4ae686b (table rebalancer_dynamic_table [id=9cc21533093646e3bf8f128856c19ccc]). Expected version 1 got 0
I20240503 13:59:39.237798 10841 tablet.cc:1722] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:39.238366 10841 tablet.cc:1620] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:39.241031 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=9cc21533093646e3bf8f128856c19ccc] alter complete (version 1)
I20240503 13:59:39.243536 10842 tablet.cc:1722] T 98e8b7edc671479baeabd247d4ae686b P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:39.244069 10842 tablet.cc:1620] T 98e8b7edc671479baeabd247d4ae686b P 5f46e19644fd4f4b825c86ba6af768da: MemRowSet was empty: no flush needed.
I20240503 13:59:39.246209 6379 catalog_manager.cc:5576] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021) does not have the latest schema for tablet 98e8b7edc671479baeabd247d4ae686b (table rebalancer_dynamic_table [id=9cc21533093646e3bf8f128856c19ccc]). Expected version 1 got 0
I20240503 13:59:39.342608 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:39.347491 10841 tablet.cc:1722] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:39.347815 10841 tablet.cc:1620] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:39.348099 10795 tablet.cc:1722] T 98e8b7edc671479baeabd247d4ae686b P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:39.348366 10795 tablet.cc:1620] T 98e8b7edc671479baeabd247d4ae686b P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:39.352044 10842 tablet.cc:1722] T 98e8b7edc671479baeabd247d4ae686b P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:39.352416 10842 tablet.cc:1620] T 98e8b7edc671479baeabd247d4ae686b P 5f46e19644fd4f4b825c86ba6af768da: MemRowSet was empty: no flush needed.
I20240503 13:59:39.352736 6386 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=9cc21533093646e3bf8f128856c19ccc] alter complete (version 2)
I20240503 13:59:39.371404 10769 raft_consensus.cc:482] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:39.371523 10769 raft_consensus.cc:504] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:39.371980 10769 leader_election.cc:288] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:39.372367 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98466061b550450bb9d0041427f6f19d" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:39.372419 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98466061b550450bb9d0041427f6f19d" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:39.372515 9343 raft_consensus.cc:2454] T 98466061b550450bb9d0041427f6f19d P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:39.372534 9081 raft_consensus.cc:2454] T 98466061b550450bb9d0041427f6f19d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:39.372834 8884 leader_election.cc:302] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 0d77752181e944698e809e325e5937e6; no voters:
I20240503 13:59:39.373040 10769 raft_consensus.cc:2781] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:39.373098 10769 raft_consensus.cc:482] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:39.373152 10769 raft_consensus.cc:3037] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:39.373967 10769 raft_consensus.cc:504] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:39.374255 10769 leader_election.cc:288] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:39.374547 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98466061b550450bb9d0041427f6f19d" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:39.374646 9081 raft_consensus.cc:3037] T 98466061b550450bb9d0041427f6f19d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:39.374714 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98466061b550450bb9d0041427f6f19d" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:39.374821 9343 raft_consensus.cc:3037] T 98466061b550450bb9d0041427f6f19d P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:39.375403 9081 raft_consensus.cc:2454] T 98466061b550450bb9d0041427f6f19d P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:39.375576 9343 raft_consensus.cc:2454] T 98466061b550450bb9d0041427f6f19d P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:39.375656 8883 leader_election.cc:302] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:39.375828 10769 raft_consensus.cc:2781] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:39.375943 10769 raft_consensus.cc:686] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 13:59:39.376082 10769 consensus_queue.cc:237] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:39.377941 6386 catalog_manager.cc:5484] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:39.387801 10751 raft_consensus.cc:482] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:39.387908 10751 raft_consensus.cc:504] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:39.388387 10751 leader_election.cc:288] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:39.388751 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac9791abf2b4a60ab7a6e4e2d4b0de6" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:39.388887 6790 raft_consensus.cc:2454] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:39.389171 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac9791abf2b4a60ab7a6e4e2d4b0de6" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:39.389307 6527 raft_consensus.cc:2454] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:39.389470 9277 leader_election.cc:302] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:39.389672 10751 raft_consensus.cc:2781] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:39.389737 10751 raft_consensus.cc:482] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:39.389784 10751 raft_consensus.cc:3037] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:39.390645 10751 raft_consensus.cc:504] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:39.390964 10751 leader_election.cc:288] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:39.391232 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac9791abf2b4a60ab7a6e4e2d4b0de6" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:39.391275 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac9791abf2b4a60ab7a6e4e2d4b0de6" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:39.391398 6790 raft_consensus.cc:3037] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:39.391408 6527 raft_consensus.cc:3037] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:39.392248 6527 raft_consensus.cc:2454] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:39.392458 6790 raft_consensus.cc:2454] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:39.392537 9278 leader_election.cc:302] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:39.392724 10751 raft_consensus.cc:2781] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:39.392793 10751 raft_consensus.cc:686] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:39.392918 10751 consensus_queue.cc:237] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:39.394829 6386 catalog_manager.cc:5484] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:39.444337 10564 raft_consensus.cc:482] T e76ded33d780455d963c51a253cbd0e0 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:39.444435 10564 raft_consensus.cc:504] T e76ded33d780455d963c51a253cbd0e0 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:39.444859 10564 leader_election.cc:288] T e76ded33d780455d963c51a253cbd0e0 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:39.445117 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e76ded33d780455d963c51a253cbd0e0" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:39.445207 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e76ded33d780455d963c51a253cbd0e0" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:39.445259 9081 raft_consensus.cc:2454] T e76ded33d780455d963c51a253cbd0e0 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:39.445386 6660 raft_consensus.cc:2454] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:39.445785 9279 leader_election.cc:302] T e76ded33d780455d963c51a253cbd0e0 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 2145d15ab2d74c0287839266f1411a43; no voters:
I20240503 13:59:39.446084 10564 raft_consensus.cc:2781] T e76ded33d780455d963c51a253cbd0e0 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:39.446148 10564 raft_consensus.cc:482] T e76ded33d780455d963c51a253cbd0e0 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:39.446197 10564 raft_consensus.cc:3037] T e76ded33d780455d963c51a253cbd0e0 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:39.454515 6386 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:39.459734 10841 tablet.cc:1722] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:39.460446 10795 tablet.cc:1722] T 98e8b7edc671479baeabd247d4ae686b P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:39.460974 10842 tablet.cc:1722] T 98e8b7edc671479baeabd247d4ae686b P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:39.466544 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=9cc21533093646e3bf8f128856c19ccc] alter complete (version 3)
I20240503 13:59:39.468233 10564 raft_consensus.cc:504] T e76ded33d780455d963c51a253cbd0e0 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:39.468914 10564 leader_election.cc:288] T e76ded33d780455d963c51a253cbd0e0 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:39.469198 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e76ded33d780455d963c51a253cbd0e0" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:39.469312 9081 raft_consensus.cc:3037] T e76ded33d780455d963c51a253cbd0e0 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:39.470250 9081 raft_consensus.cc:2454] T e76ded33d780455d963c51a253cbd0e0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:39.470539 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e76ded33d780455d963c51a253cbd0e0" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:39.470646 6660 raft_consensus.cc:3037] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:39.471698 6660 raft_consensus.cc:2454] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:39.471920 9278 leader_election.cc:302] T e76ded33d780455d963c51a253cbd0e0 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:39.472306 10564 raft_consensus.cc:2781] T e76ded33d780455d963c51a253cbd0e0 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:39.472388 10564 raft_consensus.cc:686] T e76ded33d780455d963c51a253cbd0e0 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:39.472517 10564 consensus_queue.cc:237] T e76ded33d780455d963c51a253cbd0e0 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:39.474450 6380 catalog_manager.cc:5484] T e76ded33d780455d963c51a253cbd0e0 P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:39.568168 6380 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:39.568360 6380 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:39.570547 6380 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 98e8b7edc671479baeabd247d4ae686b
I20240503 13:59:39.571086 9322 tablet_service.cc:1514] Processing DeleteTablet for tablet 98e8b7edc671479baeabd247d4ae686b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:39 UTC) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:39.571205 8928 tablet_service.cc:1514] Processing DeleteTablet for tablet 98e8b7edc671479baeabd247d4ae686b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:39 UTC) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:39.571231 6508 tablet_service.cc:1514] Processing DeleteTablet for tablet 98e8b7edc671479baeabd247d4ae686b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:39 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:39.571477 10848 tablet_replica.cc:330] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:39.571565 10849 tablet_replica.cc:330] T 98e8b7edc671479baeabd247d4ae686b P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:39.571579 10850 tablet_replica.cc:330] T 98e8b7edc671479baeabd247d4ae686b P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:39.571686 10848 raft_consensus.cc:2229] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:39.571723 10850 raft_consensus.cc:2229] T 98e8b7edc671479baeabd247d4ae686b P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:39.571777 10849 raft_consensus.cc:2229] T 98e8b7edc671479baeabd247d4ae686b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:39.571841 10850 raft_consensus.cc:2258] T 98e8b7edc671479baeabd247d4ae686b P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:39.571929 10849 raft_consensus.cc:2258] T 98e8b7edc671479baeabd247d4ae686b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:39.572306 10848 raft_consensus.cc:2258] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:39.572535 10850 ts_tablet_manager.cc:1875] T 98e8b7edc671479baeabd247d4ae686b P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:39.572535 10849 ts_tablet_manager.cc:1875] T 98e8b7edc671479baeabd247d4ae686b P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:39.572815 10848 ts_tablet_manager.cc:1875] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:39.574893 10849 ts_tablet_manager.cc:1888] T 98e8b7edc671479baeabd247d4ae686b P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:39.574903 10848 ts_tablet_manager.cc:1888] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:39.574970 10849 log.cc:1196] T 98e8b7edc671479baeabd247d4ae686b P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/98e8b7edc671479baeabd247d4ae686b
I20240503 13:59:39.574990 10848 log.cc:1196] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/98e8b7edc671479baeabd247d4ae686b
I20240503 13:59:39.575335 10849 ts_tablet_manager.cc:1909] T 98e8b7edc671479baeabd247d4ae686b P 06ca8dbc004c42b99faa527a4e6f9545: Deleting consensus metadata
I20240503 13:59:39.575338 10848 ts_tablet_manager.cc:1909] T 98e8b7edc671479baeabd247d4ae686b P 0d77752181e944698e809e325e5937e6: Deleting consensus metadata
I20240503 13:59:39.575350 10850 ts_tablet_manager.cc:1888] T 98e8b7edc671479baeabd247d4ae686b P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:39.575415 10850 log.cc:1196] T 98e8b7edc671479baeabd247d4ae686b P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/98e8b7edc671479baeabd247d4ae686b
I20240503 13:59:39.575743 10850 ts_tablet_manager.cc:1909] T 98e8b7edc671479baeabd247d4ae686b P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:39.575966 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet 98e8b7edc671479baeabd247d4ae686b (table rebalancer_dynamic_table_new_name [id=9cc21533093646e3bf8f128856c19ccc]) successfully deleted
I20240503 13:59:39.575966 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 98e8b7edc671479baeabd247d4ae686b (table rebalancer_dynamic_table_new_name [id=9cc21533093646e3bf8f128856c19ccc]) successfully deleted
I20240503 13:59:39.576323 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 98e8b7edc671479baeabd247d4ae686b (table rebalancer_dynamic_table_new_name [id=9cc21533093646e3bf8f128856c19ccc]) successfully deleted
I20240503 13:59:39.673077 6380 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:39.677929 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 2c27a3f5b40740bb95138bf3e82c6fbe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=154c93987df240989c48080608a5cbc8]), partition=
I20240503 13:59:39.678025 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 2c27a3f5b40740bb95138bf3e82c6fbe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=154c93987df240989c48080608a5cbc8]), partition=
I20240503 13:59:39.678202 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c27a3f5b40740bb95138bf3e82c6fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:39.678290 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c27a3f5b40740bb95138bf3e82c6fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:39.680423 10824 tablet_bootstrap.cc:492] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:39.680596 9322 tablet_service.cc:1467] Processing CreateTablet for tablet 2c27a3f5b40740bb95138bf3e82c6fbe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=154c93987df240989c48080608a5cbc8]), partition=
I20240503 13:59:39.680701 10813 tablet_bootstrap.cc:492] T 2c27a3f5b40740bb95138bf3e82c6fbe P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:39.680845 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c27a3f5b40740bb95138bf3e82c6fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:39.681488 10824 tablet_bootstrap.cc:654] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:39.682396 10813 tablet_bootstrap.cc:654] T 2c27a3f5b40740bb95138bf3e82c6fbe P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:39.683027 10812 tablet_bootstrap.cc:492] T 2c27a3f5b40740bb95138bf3e82c6fbe P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:39.683182 10824 tablet_bootstrap.cc:492] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:39.683259 10824 ts_tablet_manager.cc:1367] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 13:59:39.683682 10824 raft_consensus.cc:348] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:39.683794 10824 raft_consensus.cc:374] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:39.683869 10824 raft_consensus.cc:729] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:39.683879 10812 tablet_bootstrap.cc:654] T 2c27a3f5b40740bb95138bf3e82c6fbe P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:39.684010 10824 consensus_queue.cc:260] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:39.684342 10824 ts_tablet_manager.cc:1398] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:39.686332 10812 tablet_bootstrap.cc:492] T 2c27a3f5b40740bb95138bf3e82c6fbe P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:39.686389 10813 tablet_bootstrap.cc:492] T 2c27a3f5b40740bb95138bf3e82c6fbe P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:39.686417 10812 ts_tablet_manager.cc:1367] T 2c27a3f5b40740bb95138bf3e82c6fbe P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:59:39.686480 10813 ts_tablet_manager.cc:1367] T 2c27a3f5b40740bb95138bf3e82c6fbe P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 13:59:39.686789 10812 raft_consensus.cc:348] T 2c27a3f5b40740bb95138bf3e82c6fbe P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:39.686897 10812 raft_consensus.cc:374] T 2c27a3f5b40740bb95138bf3e82c6fbe P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:39.686936 10812 raft_consensus.cc:729] T 2c27a3f5b40740bb95138bf3e82c6fbe P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:39.687040 10812 consensus_queue.cc:260] T 2c27a3f5b40740bb95138bf3e82c6fbe P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:39.687139 10813 raft_consensus.cc:348] T 2c27a3f5b40740bb95138bf3e82c6fbe P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:39.687309 10812 ts_tablet_manager.cc:1398] T 2c27a3f5b40740bb95138bf3e82c6fbe P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:39.687243 10813 raft_consensus.cc:374] T 2c27a3f5b40740bb95138bf3e82c6fbe P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:39.687934 10813 raft_consensus.cc:729] T 2c27a3f5b40740bb95138bf3e82c6fbe P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:39.688091 10813 consensus_queue.cc:260] T 2c27a3f5b40740bb95138bf3e82c6fbe P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:39.688390 10813 ts_tablet_manager.cc:1398] T 2c27a3f5b40740bb95138bf3e82c6fbe P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:39.770958 10769 consensus_queue.cc:1035] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:39.774251 10769 consensus_queue.cc:1035] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:39.786392 10624 raft_consensus.cc:482] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:39.786505 10624 raft_consensus.cc:504] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:39.786971 10624 leader_election.cc:288] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:39.787359 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c27a3f5b40740bb95138bf3e82c6fbe" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:39.787405 9342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c27a3f5b40740bb95138bf3e82c6fbe" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:39.787520 9342 raft_consensus.cc:2454] T 2c27a3f5b40740bb95138bf3e82c6fbe P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:39.787534 8948 raft_consensus.cc:2454] T 2c27a3f5b40740bb95138bf3e82c6fbe P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:39.787853 9016 leader_election.cc:302] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:39.788149 10624 raft_consensus.cc:2781] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:39.788240 10624 raft_consensus.cc:482] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:39.788341 10624 raft_consensus.cc:3037] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:39.789383 10624 raft_consensus.cc:504] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:39.789716 10624 leader_election.cc:288] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:39.790110 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c27a3f5b40740bb95138bf3e82c6fbe" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:39.790232 8948 raft_consensus.cc:3037] T 2c27a3f5b40740bb95138bf3e82c6fbe P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:39.790409 9342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c27a3f5b40740bb95138bf3e82c6fbe" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:39.790536 9342 raft_consensus.cc:3037] T 2c27a3f5b40740bb95138bf3e82c6fbe P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:39.791074 8948 raft_consensus.cc:2454] T 2c27a3f5b40740bb95138bf3e82c6fbe P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:39.791375 9016 leader_election.cc:302] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:39.791608 10624 raft_consensus.cc:2781] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:39.791663 9342 raft_consensus.cc:2454] T 2c27a3f5b40740bb95138bf3e82c6fbe P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:39.791805 10624 raft_consensus.cc:686] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 13:59:39.791967 10624 consensus_queue.cc:237] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:39.794158 6379 catalog_manager.cc:5484] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:39.884712 10751 consensus_queue.cc:1035] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:39.888232 10846 consensus_queue.cc:1035] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:39.907455 6386 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:39.912577 10751 consensus_queue.cc:1035] T e76ded33d780455d963c51a253cbd0e0 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:39.912822 9342 raft_consensus.cc:1261] T 2c27a3f5b40740bb95138bf3e82c6fbe P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:39.913151 8947 raft_consensus.cc:1261] T 2c27a3f5b40740bb95138bf3e82c6fbe P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:39.914556 10624 consensus_queue.cc:1035] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:39.917650 10624 consensus_queue.cc:1035] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:39.918344 6379 catalog_manager.cc:5576] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931) does not have the latest schema for tablet 2c27a3f5b40740bb95138bf3e82c6fbe (table rebalancer_dynamic_table [id=154c93987df240989c48080608a5cbc8]). Expected version 1 got 0
I20240503 13:59:39.920444 10846 consensus_queue.cc:1035] T e76ded33d780455d963c51a253cbd0e0 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:39.920770 10856 mvcc.cc:205] Tried to move back new op lower bound from 7023594618519130112 to 7023594618029142016. Current Snapshot: MvccSnapshot[applied={T|T < 7023594618519130112}]
I20240503 13:59:39.921658 6386 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet 2c27a3f5b40740bb95138bf3e82c6fbe (table rebalancer_dynamic_table [id=154c93987df240989c48080608a5cbc8]). Expected version 1 got 0
I20240503 13:59:39.923395 6386 catalog_manager.cc:5576] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481) does not have the latest schema for tablet 2c27a3f5b40740bb95138bf3e82c6fbe (table rebalancer_dynamic_table [id=154c93987df240989c48080608a5cbc8]). Expected version 1 got 0
I20240503 13:59:39.924206 10795 tablet.cc:1722] T 2c27a3f5b40740bb95138bf3e82c6fbe P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:39.924333 10841 tablet.cc:1722] T 2c27a3f5b40740bb95138bf3e82c6fbe P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:39.924528 10795 tablet.cc:1620] T 2c27a3f5b40740bb95138bf3e82c6fbe P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:39.924572 10841 tablet.cc:1620] T 2c27a3f5b40740bb95138bf3e82c6fbe P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:39.928473 10867 tablet.cc:1722] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:39.928779 10867 tablet.cc:1620] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:39.928895 6379 catalog_manager.cc:5576] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481) does not have the latest schema for tablet 2c27a3f5b40740bb95138bf3e82c6fbe (table rebalancer_dynamic_table [id=154c93987df240989c48080608a5cbc8]). Expected version 1 got 0
I20240503 13:59:39.932623 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=154c93987df240989c48080608a5cbc8] alter complete (version 1)
I20240503 13:59:39.995360 6379 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:40.002969 9322 tablet_service.cc:1467] Processing CreateTablet for tablet 3d9906a118744dd6a78d50e52f1961e2 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=17d62e19cb344d018bc5ae78ffca2185]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:40.003315 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d9906a118744dd6a78d50e52f1961e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.003624 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 3d9906a118744dd6a78d50e52f1961e2 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=17d62e19cb344d018bc5ae78ffca2185]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:40.003902 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d9906a118744dd6a78d50e52f1961e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.005470 10812 tablet_bootstrap.cc:492] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:40.005812 10869 tablet_bootstrap.cc:492] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:40.006461 10812 tablet_bootstrap.cc:654] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:40.007316 10869 tablet_bootstrap.cc:654] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:40.007678 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 956707b321b248e3822226c44c406b8b (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=17d62e19cb344d018bc5ae78ffca2185]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:40.007871 6902 tablet_service.cc:1467] Processing CreateTablet for tablet 3d9906a118744dd6a78d50e52f1961e2 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=17d62e19cb344d018bc5ae78ffca2185]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:40.008008 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 956707b321b248e3822226c44c406b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.008266 6902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d9906a118744dd6a78d50e52f1961e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.008432 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 956707b321b248e3822226c44c406b8b (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=17d62e19cb344d018bc5ae78ffca2185]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:40.008689 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 956707b321b248e3822226c44c406b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.010810 6901 tablet_service.cc:1467] Processing CreateTablet for tablet 956707b321b248e3822226c44c406b8b (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=17d62e19cb344d018bc5ae78ffca2185]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:40.010949 6900 tablet_service.cc:1467] Processing CreateTablet for tablet 218b25613baf48319576e433c2ac4534 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=17d62e19cb344d018bc5ae78ffca2185]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:40.011065 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 956707b321b248e3822226c44c406b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.011107 10872 tablet_bootstrap.cc:492] T 3d9906a118744dd6a78d50e52f1961e2 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:40.011058 8926 tablet_service.cc:1467] Processing CreateTablet for tablet 218b25613baf48319576e433c2ac4534 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=17d62e19cb344d018bc5ae78ffca2185]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:40.011345 8926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 218b25613baf48319576e433c2ac4534. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.011243 6900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 218b25613baf48319576e433c2ac4534. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.012005 10872 tablet_bootstrap.cc:654] T 3d9906a118744dd6a78d50e52f1961e2 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:40.013706 10813 tablet_bootstrap.cc:492] T 956707b321b248e3822226c44c406b8b P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:40.014025 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 218b25613baf48319576e433c2ac4534 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=17d62e19cb344d018bc5ae78ffca2185]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:40.014295 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 218b25613baf48319576e433c2ac4534. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.014686 10813 tablet_bootstrap.cc:654] T 956707b321b248e3822226c44c406b8b P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:40.016723 10871 tablet_bootstrap.cc:492] T 956707b321b248e3822226c44c406b8b P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:40.017974 10871 tablet_bootstrap.cc:654] T 956707b321b248e3822226c44c406b8b P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:40.018152 10813 tablet_bootstrap.cc:492] T 956707b321b248e3822226c44c406b8b P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:40.018265 10813 ts_tablet_manager.cc:1367] T 956707b321b248e3822226c44c406b8b P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240503 13:59:40.018721 10813 raft_consensus.cc:348] T 956707b321b248e3822226c44c406b8b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:40.018834 10813 raft_consensus.cc:374] T 956707b321b248e3822226c44c406b8b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:40.019061 10813 raft_consensus.cc:729] T 956707b321b248e3822226c44c406b8b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:40.019188 10871 tablet_bootstrap.cc:492] T 956707b321b248e3822226c44c406b8b P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:40.019325 10871 ts_tablet_manager.cc:1367] T 956707b321b248e3822226c44c406b8b P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:40.019928 10812 tablet_bootstrap.cc:492] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:40.019870 10871 raft_consensus.cc:348] T 956707b321b248e3822226c44c406b8b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:40.019984 10871 raft_consensus.cc:374] T 956707b321b248e3822226c44c406b8b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:40.020023 10871 raft_consensus.cc:729] T 956707b321b248e3822226c44c406b8b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:40.020025 10812 ts_tablet_manager.cc:1367] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 13:59:40.019995 10813 consensus_queue.cc:260] T 956707b321b248e3822226c44c406b8b P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:40.020144 10871 consensus_queue.cc:260] T 956707b321b248e3822226c44c406b8b P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:40.020479 10871 ts_tablet_manager.cc:1398] T 956707b321b248e3822226c44c406b8b P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:40.020505 10812 raft_consensus.cc:348] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:40.020602 10812 raft_consensus.cc:374] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:40.020635 10812 raft_consensus.cc:729] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:40.020750 10812 consensus_queue.cc:260] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:40.021003 10812 ts_tablet_manager.cc:1398] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:40.021147 10813 ts_tablet_manager.cc:1398] T 956707b321b248e3822226c44c406b8b P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:59:40.021347 10813 tablet_bootstrap.cc:492] T 218b25613baf48319576e433c2ac4534 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:40.022387 10813 tablet_bootstrap.cc:654] T 218b25613baf48319576e433c2ac4534 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:40.024708 10872 tablet_bootstrap.cc:492] T 3d9906a118744dd6a78d50e52f1961e2 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:40.024824 10872 ts_tablet_manager.cc:1367] T 3d9906a118744dd6a78d50e52f1961e2 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 13:59:40.025362 10872 raft_consensus.cc:348] T 3d9906a118744dd6a78d50e52f1961e2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:40.025465 10872 raft_consensus.cc:374] T 3d9906a118744dd6a78d50e52f1961e2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:40.025506 10872 raft_consensus.cc:729] T 3d9906a118744dd6a78d50e52f1961e2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:40.025624 10872 consensus_queue.cc:260] T 3d9906a118744dd6a78d50e52f1961e2 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:40.025827 10869 tablet_bootstrap.cc:492] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:40.025926 10872 ts_tablet_manager.cc:1398] T 3d9906a118744dd6a78d50e52f1961e2 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:40.026077 10872 tablet_bootstrap.cc:492] T 956707b321b248e3822226c44c406b8b P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:40.026226 10869 ts_tablet_manager.cc:1367] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 13:59:40.026729 10869 raft_consensus.cc:348] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:40.026875 10869 raft_consensus.cc:374] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:40.026999 10872 tablet_bootstrap.cc:654] T 956707b321b248e3822226c44c406b8b P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:40.026909 10869 raft_consensus.cc:729] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:40.029775 10869 consensus_queue.cc:260] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:40.030210 10869 ts_tablet_manager.cc:1398] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:59:40.030375 10869 tablet_bootstrap.cc:492] T 218b25613baf48319576e433c2ac4534 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:40.031342 10869 tablet_bootstrap.cc:654] T 218b25613baf48319576e433c2ac4534 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:40.032600 10872 tablet_bootstrap.cc:492] T 956707b321b248e3822226c44c406b8b P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:40.032699 10872 ts_tablet_manager.cc:1367] T 956707b321b248e3822226c44c406b8b P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:59:40.033268 10872 raft_consensus.cc:348] T 956707b321b248e3822226c44c406b8b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:40.033449 10872 raft_consensus.cc:374] T 956707b321b248e3822226c44c406b8b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:40.033521 10872 raft_consensus.cc:729] T 956707b321b248e3822226c44c406b8b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:40.033659 10872 consensus_queue.cc:260] T 956707b321b248e3822226c44c406b8b P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:40.033958 10872 ts_tablet_manager.cc:1398] T 956707b321b248e3822226c44c406b8b P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:40.034168 10872 tablet_bootstrap.cc:492] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:40.035619 10872 tablet_bootstrap.cc:654] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:40.036144 10564 raft_consensus.cc:482] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:40.036294 10564 raft_consensus.cc:504] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:40.036852 10564 leader_election.cc:288] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:40.037041 10872 tablet_bootstrap.cc:492] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:40.037117 10872 ts_tablet_manager.cc:1367] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:40.037412 10869 tablet_bootstrap.cc:492] T 218b25613baf48319576e433c2ac4534 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:40.037499 10869 ts_tablet_manager.cc:1367] T 218b25613baf48319576e433c2ac4534 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 13:59:40.037616 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d9906a118744dd6a78d50e52f1961e2" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:40.037720 6920 raft_consensus.cc:2454] T 3d9906a118744dd6a78d50e52f1961e2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:40.037685 10872 raft_consensus.cc:348] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:40.037797 10872 raft_consensus.cc:374] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:40.037850 10872 raft_consensus.cc:729] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:40.038005 9278 leader_election.cc:302] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:40.037966 10872 consensus_queue.cc:260] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:40.038220 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d9906a118744dd6a78d50e52f1961e2" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:40.038359 10846 raft_consensus.cc:2781] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:40.038362 6527 raft_consensus.cc:2454] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:40.038410 10846 raft_consensus.cc:482] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:40.038447 10846 raft_consensus.cc:3037] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:40.038443 10872 ts_tablet_manager.cc:1398] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:40.038619 10869 raft_consensus.cc:348] T 218b25613baf48319576e433c2ac4534 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:40.038722 10869 raft_consensus.cc:374] T 218b25613baf48319576e433c2ac4534 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:40.038753 10869 raft_consensus.cc:729] T 218b25613baf48319576e433c2ac4534 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:40.038872 10869 consensus_queue.cc:260] T 218b25613baf48319576e433c2ac4534 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:40.039218 10869 ts_tablet_manager.cc:1398] T 218b25613baf48319576e433c2ac4534 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 13:59:40.039654 10813 tablet_bootstrap.cc:492] T 218b25613baf48319576e433c2ac4534 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:40.039597 10846 raft_consensus.cc:504] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:40.039765 10813 ts_tablet_manager.cc:1367] T 218b25613baf48319576e433c2ac4534 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.002s
I20240503 13:59:40.039954 10846 leader_election.cc:288] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:40.040144 10813 raft_consensus.cc:348] T 218b25613baf48319576e433c2ac4534 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:40.040241 10813 raft_consensus.cc:374] T 218b25613baf48319576e433c2ac4534 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:40.040264 10813 raft_consensus.cc:729] T 218b25613baf48319576e433c2ac4534 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:40.040355 10813 consensus_queue.cc:260] T 218b25613baf48319576e433c2ac4534 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:40.040396 6386 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:40.040609 10813 ts_tablet_manager.cc:1398] T 218b25613baf48319576e433c2ac4534 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:40.041406 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d9906a118744dd6a78d50e52f1961e2" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:40.041525 6920 raft_consensus.cc:3037] T 3d9906a118744dd6a78d50e52f1961e2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:40.042560 6920 raft_consensus.cc:2454] T 3d9906a118744dd6a78d50e52f1961e2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:40.042920 9278 leader_election.cc:302] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:40.043157 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d9906a118744dd6a78d50e52f1961e2" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:40.043294 6527 raft_consensus.cc:3037] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:40.043569 10846 raft_consensus.cc:2781] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:40.043645 10846 raft_consensus.cc:686] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:40.043779 10846 consensus_queue.cc:237] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:40.044307 6527 raft_consensus.cc:2454] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:40.046167 6383 catalog_manager.cc:5484] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:40.052162 10867 tablet.cc:1722] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:40.052549 10867 tablet.cc:1620] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:40.053647 10795 tablet.cc:1722] T 2c27a3f5b40740bb95138bf3e82c6fbe P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:40.053663 10841 tablet.cc:1722] T 2c27a3f5b40740bb95138bf3e82c6fbe P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:40.053979 10841 tablet.cc:1620] T 2c27a3f5b40740bb95138bf3e82c6fbe P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:40.054006 10795 tablet.cc:1620] T 2c27a3f5b40740bb95138bf3e82c6fbe P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:40.059206 6386 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=154c93987df240989c48080608a5cbc8] alter complete (version 2)
I20240503 13:59:40.116705 10731 raft_consensus.cc:482] T 956707b321b248e3822226c44c406b8b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:40.116838 10731 raft_consensus.cc:504] T 956707b321b248e3822226c44c406b8b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:40.117300 10731 leader_election.cc:288] T 956707b321b248e3822226c44c406b8b P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:40.117573 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "956707b321b248e3822226c44c406b8b" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:40.117651 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "956707b321b248e3822226c44c406b8b" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:40.117728 6922 raft_consensus.cc:2454] T 956707b321b248e3822226c44c406b8b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:40.117807 6660 raft_consensus.cc:2454] T 956707b321b248e3822226c44c406b8b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:40.118067 8883 leader_election.cc:302] T 956707b321b248e3822226c44c406b8b P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:40.118237 10731 raft_consensus.cc:2781] T 956707b321b248e3822226c44c406b8b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:40.118295 10731 raft_consensus.cc:482] T 956707b321b248e3822226c44c406b8b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:40.118341 10731 raft_consensus.cc:3037] T 956707b321b248e3822226c44c406b8b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:40.119182 10731 raft_consensus.cc:504] T 956707b321b248e3822226c44c406b8b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:40.119519 10731 leader_election.cc:288] T 956707b321b248e3822226c44c406b8b P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:40.119879 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "956707b321b248e3822226c44c406b8b" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:40.119882 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "956707b321b248e3822226c44c406b8b" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:40.119997 6922 raft_consensus.cc:3037] T 956707b321b248e3822226c44c406b8b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:40.120000 6660 raft_consensus.cc:3037] T 956707b321b248e3822226c44c406b8b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:40.120748 6660 raft_consensus.cc:2454] T 956707b321b248e3822226c44c406b8b P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:40.120941 6922 raft_consensus.cc:2454] T 956707b321b248e3822226c44c406b8b P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:40.121009 8884 leader_election.cc:302] T 956707b321b248e3822226c44c406b8b P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 2145d15ab2d74c0287839266f1411a43; no voters:
I20240503 13:59:40.121158 10731 raft_consensus.cc:2781] T 956707b321b248e3822226c44c406b8b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:40.121234 10731 raft_consensus.cc:686] T 956707b321b248e3822226c44c406b8b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 13:59:40.121371 10731 consensus_queue.cc:237] T 956707b321b248e3822226c44c406b8b P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:40.123438 6383 catalog_manager.cc:5484] T 956707b321b248e3822226c44c406b8b P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:40.162438 10883 raft_consensus.cc:482] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:40.162554 10883 raft_consensus.cc:504] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:40.163023 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:40.163477 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218b25613baf48319576e433c2ac4534" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:40.163597 8947 raft_consensus.cc:2454] T 218b25613baf48319576e433c2ac4534 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:40.163926 6857 leader_election.cc:302] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:40.164096 10507 raft_consensus.cc:2781] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:40.164115 10883 leader_election.cc:288] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:40.164155 10507 raft_consensus.cc:482] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:40.164216 10507 raft_consensus.cc:3037] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:40.164539 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218b25613baf48319576e433c2ac4534" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:40.164664 6527 raft_consensus.cc:2454] T 218b25613baf48319576e433c2ac4534 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:40.165222 10507 raft_consensus.cc:504] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:40.165524 10507 leader_election.cc:288] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:40.165977 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218b25613baf48319576e433c2ac4534" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:40.166070 6527 raft_consensus.cc:3037] T 218b25613baf48319576e433c2ac4534 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:40.167050 6527 raft_consensus.cc:2454] T 218b25613baf48319576e433c2ac4534 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:40.167369 6857 leader_election.cc:302] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:40.167520 10507 raft_consensus.cc:2781] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:40.167596 10507 raft_consensus.cc:686] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:40.167735 10507 consensus_queue.cc:237] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:40.169457 10867 tablet.cc:1722] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:40.170065 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218b25613baf48319576e433c2ac4534" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:40.170181 8947 raft_consensus.cc:3037] T 218b25613baf48319576e433c2ac4534 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:40.170490 6383 catalog_manager.cc:5484] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:40.171186 8947 raft_consensus.cc:2454] T 218b25613baf48319576e433c2ac4534 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:40.172005 10841 tablet.cc:1722] T 2c27a3f5b40740bb95138bf3e82c6fbe P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:40.172621 10795 tablet.cc:1722] T 2c27a3f5b40740bb95138bf3e82c6fbe P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:40.177696 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=154c93987df240989c48080608a5cbc8] alter complete (version 3)
I20240503 13:59:40.279799 6386 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:40.279945 6386 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:40.281852 6386 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 2c27a3f5b40740bb95138bf3e82c6fbe
I20240503 13:59:40.282446 9058 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c27a3f5b40740bb95138bf3e82c6fbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:40 UTC) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:40.282557 8926 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c27a3f5b40740bb95138bf3e82c6fbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:40 UTC) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:40.282575 9322 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c27a3f5b40740bb95138bf3e82c6fbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:40 UTC) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:40.282836 10887 tablet_replica.cc:330] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:40.282907 10888 tablet_replica.cc:330] T 2c27a3f5b40740bb95138bf3e82c6fbe P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:40.282953 10889 tablet_replica.cc:330] T 2c27a3f5b40740bb95138bf3e82c6fbe P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:40.282999 10887 raft_consensus.cc:2229] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:40.283084 10888 raft_consensus.cc:2229] T 2c27a3f5b40740bb95138bf3e82c6fbe P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:40.283183 10889 raft_consensus.cc:2229] T 2c27a3f5b40740bb95138bf3e82c6fbe P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:40.283215 10887 raft_consensus.cc:2258] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:40.283234 10888 raft_consensus.cc:2258] T 2c27a3f5b40740bb95138bf3e82c6fbe P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:40.283334 10889 raft_consensus.cc:2258] T 2c27a3f5b40740bb95138bf3e82c6fbe P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:40.283864 10888 ts_tablet_manager.cc:1875] T 2c27a3f5b40740bb95138bf3e82c6fbe P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:40.283912 10889 ts_tablet_manager.cc:1875] T 2c27a3f5b40740bb95138bf3e82c6fbe P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:40.283967 10887 ts_tablet_manager.cc:1875] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:40.285862 10888 ts_tablet_manager.cc:1888] T 2c27a3f5b40740bb95138bf3e82c6fbe P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:40.285966 10888 log.cc:1196] T 2c27a3f5b40740bb95138bf3e82c6fbe P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/2c27a3f5b40740bb95138bf3e82c6fbe
I20240503 13:59:40.286069 10889 ts_tablet_manager.cc:1888] T 2c27a3f5b40740bb95138bf3e82c6fbe P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:40.286069 10887 ts_tablet_manager.cc:1888] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:40.286156 10887 log.cc:1196] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/2c27a3f5b40740bb95138bf3e82c6fbe
I20240503 13:59:40.286170 10889 log.cc:1196] T 2c27a3f5b40740bb95138bf3e82c6fbe P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/2c27a3f5b40740bb95138bf3e82c6fbe
I20240503 13:59:40.286293 10888 ts_tablet_manager.cc:1909] T 2c27a3f5b40740bb95138bf3e82c6fbe P 06ca8dbc004c42b99faa527a4e6f9545: Deleting consensus metadata
I20240503 13:59:40.286478 10889 ts_tablet_manager.cc:1909] T 2c27a3f5b40740bb95138bf3e82c6fbe P 0d77752181e944698e809e325e5937e6: Deleting consensus metadata
I20240503 13:59:40.286530 10887 ts_tablet_manager.cc:1909] T 2c27a3f5b40740bb95138bf3e82c6fbe P 8d574fb4a89d4eea947c859470d44dd4: Deleting consensus metadata
I20240503 13:59:40.287134 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 2c27a3f5b40740bb95138bf3e82c6fbe (table rebalancer_dynamic_table_new_name [id=154c93987df240989c48080608a5cbc8]) successfully deleted
I20240503 13:59:40.287724 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet 2c27a3f5b40740bb95138bf3e82c6fbe (table rebalancer_dynamic_table_new_name [id=154c93987df240989c48080608a5cbc8]) successfully deleted
I20240503 13:59:40.288043 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet 2c27a3f5b40740bb95138bf3e82c6fbe (table rebalancer_dynamic_table_new_name [id=154c93987df240989c48080608a5cbc8]) successfully deleted
I20240503 13:59:40.383479 6386 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:40.389292 9322 tablet_service.cc:1467] Processing CreateTablet for tablet 905a0cc9091b47029b02bc931c2e0fa7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2ee59d4f31b746c492f6a181301c77b0]), partition=
I20240503 13:59:40.389592 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 905a0cc9091b47029b02bc931c2e0fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.391336 10812 tablet_bootstrap.cc:492] T 905a0cc9091b47029b02bc931c2e0fa7 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:40.391669 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 905a0cc9091b47029b02bc931c2e0fa7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2ee59d4f31b746c492f6a181301c77b0]), partition=
I20240503 13:59:40.391947 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 905a0cc9091b47029b02bc931c2e0fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.392094 9190 tablet_service.cc:1467] Processing CreateTablet for tablet 905a0cc9091b47029b02bc931c2e0fa7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2ee59d4f31b746c492f6a181301c77b0]), partition=
I20240503 13:59:40.392289 10812 tablet_bootstrap.cc:654] T 905a0cc9091b47029b02bc931c2e0fa7 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:40.392315 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 905a0cc9091b47029b02bc931c2e0fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.393558 10812 tablet_bootstrap.cc:492] T 905a0cc9091b47029b02bc931c2e0fa7 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:40.393649 10812 ts_tablet_manager.cc:1367] T 905a0cc9091b47029b02bc931c2e0fa7 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:40.394111 10812 raft_consensus.cc:348] T 905a0cc9091b47029b02bc931c2e0fa7 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:40.394227 10812 raft_consensus.cc:374] T 905a0cc9091b47029b02bc931c2e0fa7 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:40.394263 10869 tablet_bootstrap.cc:492] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:40.394251 10891 tablet_bootstrap.cc:492] T 905a0cc9091b47029b02bc931c2e0fa7 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:40.394284 10812 raft_consensus.cc:729] T 905a0cc9091b47029b02bc931c2e0fa7 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:40.394392 10812 consensus_queue.cc:260] T 905a0cc9091b47029b02bc931c2e0fa7 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:40.394683 10812 ts_tablet_manager.cc:1398] T 905a0cc9091b47029b02bc931c2e0fa7 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:40.395002 10891 tablet_bootstrap.cc:654] T 905a0cc9091b47029b02bc931c2e0fa7 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:40.395172 10869 tablet_bootstrap.cc:654] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:40.396212 10869 tablet_bootstrap.cc:492] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:40.396301 10869 ts_tablet_manager.cc:1367] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:40.396740 10869 raft_consensus.cc:348] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:40.396854 10869 raft_consensus.cc:374] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:40.396888 10869 raft_consensus.cc:729] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:40.396999 10869 consensus_queue.cc:260] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:40.397264 10869 ts_tablet_manager.cc:1398] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:40.398018 10891 tablet_bootstrap.cc:492] T 905a0cc9091b47029b02bc931c2e0fa7 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:40.398136 10891 ts_tablet_manager.cc:1367] T 905a0cc9091b47029b02bc931c2e0fa7 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:40.398583 10891 raft_consensus.cc:348] T 905a0cc9091b47029b02bc931c2e0fa7 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:40.398877 10891 raft_consensus.cc:374] T 905a0cc9091b47029b02bc931c2e0fa7 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:40.398945 10891 raft_consensus.cc:729] T 905a0cc9091b47029b02bc931c2e0fa7 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:40.399085 10891 consensus_queue.cc:260] T 905a0cc9091b47029b02bc931c2e0fa7 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:40.399434 10891 ts_tablet_manager.cc:1398] T 905a0cc9091b47029b02bc931c2e0fa7 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:40.531052 10564 consensus_queue.cc:1035] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:40.534509 10846 consensus_queue.cc:1035] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:40.559379 10862 raft_consensus.cc:482] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:40.559509 10862 raft_consensus.cc:504] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:40.559900 10862 leader_election.cc:288] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:40.560240 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "905a0cc9091b47029b02bc931c2e0fa7" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:40.560266 9342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "905a0cc9091b47029b02bc931c2e0fa7" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:40.560371 9211 raft_consensus.cc:2454] T 905a0cc9091b47029b02bc931c2e0fa7 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:40.560403 9342 raft_consensus.cc:2454] T 905a0cc9091b47029b02bc931c2e0fa7 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:40.560672 6464 leader_election.cc:302] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:40.560865 10862 raft_consensus.cc:2781] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:40.560926 10862 raft_consensus.cc:482] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:40.560963 10862 raft_consensus.cc:3037] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:40.561842 10862 raft_consensus.cc:504] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:40.562191 10862 leader_election.cc:288] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:40.562496 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "905a0cc9091b47029b02bc931c2e0fa7" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:40.562496 9342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "905a0cc9091b47029b02bc931c2e0fa7" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:40.562608 9342 raft_consensus.cc:3037] T 905a0cc9091b47029b02bc931c2e0fa7 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:40.562608 9211 raft_consensus.cc:3037] T 905a0cc9091b47029b02bc931c2e0fa7 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:40.563591 9342 raft_consensus.cc:2454] T 905a0cc9091b47029b02bc931c2e0fa7 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:40.563591 9211 raft_consensus.cc:2454] T 905a0cc9091b47029b02bc931c2e0fa7 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:40.563889 6465 leader_election.cc:302] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:40.564075 10862 raft_consensus.cc:2781] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:40.564285 10862 raft_consensus.cc:686] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Becoming Leader. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Running, Role: LEADER
I20240503 13:59:40.564441 10862 consensus_queue.cc:237] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:40.566673 6386 catalog_manager.cc:5484] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:40.570333 10886 consensus_queue.cc:1035] T 956707b321b248e3822226c44c406b8b P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:40.576601 10731 consensus_queue.cc:1035] T 956707b321b248e3822226c44c406b8b P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:40.606433 10883 consensus_queue.cc:1035] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:40.610556 10507 consensus_queue.cc:1035] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:40.679988 6380 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:40.682246 6386 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:40.686796 9211 raft_consensus.cc:1261] T 905a0cc9091b47029b02bc931c2e0fa7 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:40.687321 9342 raft_consensus.cc:1261] T 905a0cc9091b47029b02bc931c2e0fa7 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:40.687847 9058 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab63490264c4755aeab897f865e9a11 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0179e5ae38524fc9b8f37ce24d941401]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:40.687909 10898 consensus_queue.cc:1035] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:40.687834 9060 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed7d05584d14fdaafb625eaff3ecf08 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0179e5ae38524fc9b8f37ce24d941401]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:40.688177 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab63490264c4755aeab897f865e9a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.688380 6771 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed7d05584d14fdaafb625eaff3ecf08 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0179e5ae38524fc9b8f37ce24d941401]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:40.688591 9060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed7d05584d14fdaafb625eaff3ecf08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.688661 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed7d05584d14fdaafb625eaff3ecf08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.690769 10909 tablet_bootstrap.cc:492] T 2ed7d05584d14fdaafb625eaff3ecf08 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:40.687834 9061 tablet_service.cc:1467] Processing CreateTablet for tablet 4825989d826642bbb5f4bf0ea350e27c (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0179e5ae38524fc9b8f37ce24d941401]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:40.690860 6900 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab63490264c4755aeab897f865e9a11 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0179e5ae38524fc9b8f37ce24d941401]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:40.690944 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4825989d826642bbb5f4bf0ea350e27c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.691099 6900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab63490264c4755aeab897f865e9a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.691759 10909 tablet_bootstrap.cc:654] T 2ed7d05584d14fdaafb625eaff3ecf08 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:40.693094 9190 tablet_service.cc:1467] Processing CreateTablet for tablet 4825989d826642bbb5f4bf0ea350e27c (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0179e5ae38524fc9b8f37ce24d941401]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:40.693168 8926 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab63490264c4755aeab897f865e9a11 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0179e5ae38524fc9b8f37ce24d941401]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:40.693377 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4825989d826642bbb5f4bf0ea350e27c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.693483 8926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab63490264c4755aeab897f865e9a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.694237 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 4825989d826642bbb5f4bf0ea350e27c (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0179e5ae38524fc9b8f37ce24d941401]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:40.694484 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4825989d826642bbb5f4bf0ea350e27c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.695549 10891 tablet_bootstrap.cc:492] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:40.695942 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed7d05584d14fdaafb625eaff3ecf08 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0179e5ae38524fc9b8f37ce24d941401]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:40.696187 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed7d05584d14fdaafb625eaff3ecf08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.696514 10908 tablet_bootstrap.cc:492] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:40.696480 10898 consensus_queue.cc:1035] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:40.697067 10891 tablet_bootstrap.cc:654] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:40.697504 10908 tablet_bootstrap.cc:654] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:40.698200 10891 tablet_bootstrap.cc:492] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:40.698293 10891 ts_tablet_manager.cc:1367] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:40.698717 10891 raft_consensus.cc:348] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:40.698817 10891 raft_consensus.cc:374] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:40.698853 10891 raft_consensus.cc:729] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:40.698980 10891 consensus_queue.cc:260] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:40.699304 10891 ts_tablet_manager.cc:1398] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:40.700686 6386 catalog_manager.cc:5576] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931) does not have the latest schema for tablet 905a0cc9091b47029b02bc931c2e0fa7 (table rebalancer_dynamic_table [id=2ee59d4f31b746c492f6a181301c77b0]). Expected version 1 got 0
I20240503 13:59:40.701206 10909 tablet_bootstrap.cc:492] T 2ed7d05584d14fdaafb625eaff3ecf08 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:40.701318 10909 ts_tablet_manager.cc:1367] T 2ed7d05584d14fdaafb625eaff3ecf08 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 13:59:40.701349 6383 catalog_manager.cc:5576] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091) does not have the latest schema for tablet 905a0cc9091b47029b02bc931c2e0fa7 (table rebalancer_dynamic_table [id=2ee59d4f31b746c492f6a181301c77b0]). Expected version 1 got 0
I20240503 13:59:40.702415 10909 raft_consensus.cc:348] T 2ed7d05584d14fdaafb625eaff3ecf08 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:40.702605 10909 raft_consensus.cc:374] T 2ed7d05584d14fdaafb625eaff3ecf08 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:40.702749 10909 raft_consensus.cc:729] T 2ed7d05584d14fdaafb625eaff3ecf08 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:40.703671 10909 consensus_queue.cc:260] T 2ed7d05584d14fdaafb625eaff3ecf08 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:40.704795 6386 catalog_manager.cc:5576] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021) does not have the latest schema for tablet 905a0cc9091b47029b02bc931c2e0fa7 (table rebalancer_dynamic_table [id=2ee59d4f31b746c492f6a181301c77b0]). Expected version 1 got 0
I20240503 13:59:40.704821 10909 ts_tablet_manager.cc:1398] T 2ed7d05584d14fdaafb625eaff3ecf08 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:40.705077 10912 tablet_bootstrap.cc:492] T 1ab63490264c4755aeab897f865e9a11 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:40.706130 10912 tablet_bootstrap.cc:654] T 1ab63490264c4755aeab897f865e9a11 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:40.707715 10911 tablet_bootstrap.cc:492] T 1ab63490264c4755aeab897f865e9a11 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:40.708730 10911 tablet_bootstrap.cc:654] T 1ab63490264c4755aeab897f865e9a11 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:40.710130 10920 tablet.cc:1722] T 905a0cc9091b47029b02bc931c2e0fa7 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:40.710476 10912 tablet_bootstrap.cc:492] T 1ab63490264c4755aeab897f865e9a11 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:40.710696 10912 ts_tablet_manager.cc:1367] T 1ab63490264c4755aeab897f865e9a11 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:59:40.710701 10920 tablet.cc:1620] T 905a0cc9091b47029b02bc931c2e0fa7 P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:40.710695 6386 catalog_manager.cc:5576] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091) does not have the latest schema for tablet 905a0cc9091b47029b02bc931c2e0fa7 (table rebalancer_dynamic_table [id=2ee59d4f31b746c492f6a181301c77b0]). Expected version 1 got 0
I20240503 13:59:40.711329 10912 raft_consensus.cc:348] T 1ab63490264c4755aeab897f865e9a11 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:40.711499 10912 raft_consensus.cc:374] T 1ab63490264c4755aeab897f865e9a11 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:40.711628 10912 raft_consensus.cc:729] T 1ab63490264c4755aeab897f865e9a11 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:40.711812 10912 consensus_queue.cc:260] T 1ab63490264c4755aeab897f865e9a11 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:40.712191 10912 ts_tablet_manager.cc:1398] T 1ab63490264c4755aeab897f865e9a11 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:40.712817 10869 tablet_bootstrap.cc:492] T 4825989d826642bbb5f4bf0ea350e27c P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:40.712908 10911 tablet_bootstrap.cc:492] T 1ab63490264c4755aeab897f865e9a11 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:40.712983 10911 ts_tablet_manager.cc:1367] T 1ab63490264c4755aeab897f865e9a11 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:40.713439 10911 raft_consensus.cc:348] T 1ab63490264c4755aeab897f865e9a11 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:40.713536 10911 raft_consensus.cc:374] T 1ab63490264c4755aeab897f865e9a11 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:40.713565 10911 raft_consensus.cc:729] T 1ab63490264c4755aeab897f865e9a11 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:40.713691 10911 consensus_queue.cc:260] T 1ab63490264c4755aeab897f865e9a11 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:40.713780 10869 tablet_bootstrap.cc:654] T 4825989d826642bbb5f4bf0ea350e27c P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:40.714069 10911 ts_tablet_manager.cc:1398] T 1ab63490264c4755aeab897f865e9a11 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:40.714830 10912 tablet_bootstrap.cc:492] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:40.715178 10869 tablet_bootstrap.cc:492] T 4825989d826642bbb5f4bf0ea350e27c P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:40.715265 10869 ts_tablet_manager.cc:1367] T 4825989d826642bbb5f4bf0ea350e27c P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 13:59:40.715740 10869 raft_consensus.cc:348] T 4825989d826642bbb5f4bf0ea350e27c P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:40.715848 10869 raft_consensus.cc:374] T 4825989d826642bbb5f4bf0ea350e27c P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:40.715888 10869 raft_consensus.cc:729] T 4825989d826642bbb5f4bf0ea350e27c P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:40.715930 10912 tablet_bootstrap.cc:654] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:40.716019 10869 consensus_queue.cc:260] T 4825989d826642bbb5f4bf0ea350e27c P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:40.716316 6386 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=2ee59d4f31b746c492f6a181301c77b0] alter complete (version 1)
I20240503 13:59:40.716329 10869 ts_tablet_manager.cc:1398] T 4825989d826642bbb5f4bf0ea350e27c P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:40.717797 10912 tablet_bootstrap.cc:492] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:40.717921 10912 ts_tablet_manager.cc:1367] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:40.718380 10912 raft_consensus.cc:348] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:40.718521 10912 raft_consensus.cc:374] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:40.718549 10917 tablet.cc:1722] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:40.718950 10917 tablet.cc:1620] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da: MemRowSet was empty: no flush needed.
I20240503 13:59:40.720160 10912 raft_consensus.cc:729] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:40.720371 10912 consensus_queue.cc:260] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:40.720739 10912 ts_tablet_manager.cc:1398] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:40.722504 10908 tablet_bootstrap.cc:492] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:40.722662 10908 ts_tablet_manager.cc:1367] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240503 13:59:40.723629 10924 tablet.cc:1722] T 905a0cc9091b47029b02bc931c2e0fa7 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:40.724290 10908 raft_consensus.cc:348] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:40.724462 10908 raft_consensus.cc:374] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:40.724515 10908 raft_consensus.cc:729] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:40.724679 10908 consensus_queue.cc:260] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:40.725087 10908 ts_tablet_manager.cc:1398] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:40.725273 10908 tablet_bootstrap.cc:492] T 2ed7d05584d14fdaafb625eaff3ecf08 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:40.726104 10924 tablet.cc:1620] T 905a0cc9091b47029b02bc931c2e0fa7 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:40.726593 10908 tablet_bootstrap.cc:654] T 2ed7d05584d14fdaafb625eaff3ecf08 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:40.728714 10908 tablet_bootstrap.cc:492] T 2ed7d05584d14fdaafb625eaff3ecf08 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:40.728896 10908 ts_tablet_manager.cc:1367] T 2ed7d05584d14fdaafb625eaff3ecf08 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:40.730039 10908 raft_consensus.cc:348] T 2ed7d05584d14fdaafb625eaff3ecf08 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:40.730162 10908 raft_consensus.cc:374] T 2ed7d05584d14fdaafb625eaff3ecf08 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:40.730221 10908 raft_consensus.cc:729] T 2ed7d05584d14fdaafb625eaff3ecf08 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:40.730479 10908 consensus_queue.cc:260] T 2ed7d05584d14fdaafb625eaff3ecf08 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:40.731123 10908 ts_tablet_manager.cc:1398] T 2ed7d05584d14fdaafb625eaff3ecf08 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:40.731467 10908 tablet_bootstrap.cc:492] T 4825989d826642bbb5f4bf0ea350e27c P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:40.732663 10908 tablet_bootstrap.cc:654] T 4825989d826642bbb5f4bf0ea350e27c P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:40.733736 10908 tablet_bootstrap.cc:492] T 4825989d826642bbb5f4bf0ea350e27c P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:40.733839 10908 ts_tablet_manager.cc:1367] T 4825989d826642bbb5f4bf0ea350e27c P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:40.734284 10908 raft_consensus.cc:348] T 4825989d826642bbb5f4bf0ea350e27c P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:40.734405 10908 raft_consensus.cc:374] T 4825989d826642bbb5f4bf0ea350e27c P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:40.734452 10908 raft_consensus.cc:729] T 4825989d826642bbb5f4bf0ea350e27c P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:40.734589 10908 consensus_queue.cc:260] T 4825989d826642bbb5f4bf0ea350e27c P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:40.734921 10908 ts_tablet_manager.cc:1398] T 4825989d826642bbb5f4bf0ea350e27c P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:40.810050 10851 raft_consensus.cc:482] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:40.810173 10851 raft_consensus.cc:504] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:40.810591 10851 leader_election.cc:288] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:40.810917 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4825989d826642bbb5f4bf0ea350e27c" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:40.811064 9080 raft_consensus.cc:2454] T 4825989d826642bbb5f4bf0ea350e27c P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:40.811075 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4825989d826642bbb5f4bf0ea350e27c" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:40.811237 6527 raft_consensus.cc:2454] T 4825989d826642bbb5f4bf0ea350e27c P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:40.811343 9147 leader_election.cc:302] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:40.811511 10851 raft_consensus.cc:2781] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:40.811579 10851 raft_consensus.cc:482] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:40.811626 10851 raft_consensus.cc:3037] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:40.812520 10851 raft_consensus.cc:504] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:40.812816 10851 leader_election.cc:288] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:40.819669 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:40.820912 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4825989d826642bbb5f4bf0ea350e27c" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:40.820992 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4825989d826642bbb5f4bf0ea350e27c" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:40.821045 9080 raft_consensus.cc:3037] T 4825989d826642bbb5f4bf0ea350e27c P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:40.821112 6527 raft_consensus.cc:3037] T 4825989d826642bbb5f4bf0ea350e27c P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:40.822026 6527 raft_consensus.cc:2454] T 4825989d826642bbb5f4bf0ea350e27c P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:40.822039 9080 raft_consensus.cc:2454] T 4825989d826642bbb5f4bf0ea350e27c P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:40.822333 9147 leader_election.cc:302] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:40.822616 10851 raft_consensus.cc:2781] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:40.822705 10851 raft_consensus.cc:686] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 13:59:40.822826 10851 consensus_queue.cc:237] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:40.826203 10917 tablet.cc:1722] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:40.826525 10917 tablet.cc:1620] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da: MemRowSet was empty: no flush needed.
I20240503 13:59:40.826494 6379 catalog_manager.cc:5484] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:40.828126 10920 tablet.cc:1722] T 905a0cc9091b47029b02bc931c2e0fa7 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:40.828433 10920 tablet.cc:1620] T 905a0cc9091b47029b02bc931c2e0fa7 P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:40.833927 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=2ee59d4f31b746c492f6a181301c77b0] alter complete (version 2)
I20240503 13:59:40.836392 10924 tablet.cc:1722] T 905a0cc9091b47029b02bc931c2e0fa7 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:40.836827 10924 tablet.cc:1620] T 905a0cc9091b47029b02bc931c2e0fa7 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:40.910035 10731 raft_consensus.cc:482] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:40.910163 10731 raft_consensus.cc:504] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:40.910552 10731 leader_election.cc:288] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:40.910874 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed7d05584d14fdaafb625eaff3ecf08" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:40.910907 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed7d05584d14fdaafb625eaff3ecf08" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:40.911000 6791 raft_consensus.cc:2454] T 2ed7d05584d14fdaafb625eaff3ecf08 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:40.911044 9080 raft_consensus.cc:2454] T 2ed7d05584d14fdaafb625eaff3ecf08 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:40.911481 8883 leader_election.cc:302] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:40.911743 10731 raft_consensus.cc:2781] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:40.911795 10731 raft_consensus.cc:482] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:40.911832 10731 raft_consensus.cc:3037] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:40.912963 10731 raft_consensus.cc:504] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:40.913302 10731 leader_election.cc:288] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:40.913594 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed7d05584d14fdaafb625eaff3ecf08" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:40.913609 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed7d05584d14fdaafb625eaff3ecf08" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:40.913698 6791 raft_consensus.cc:3037] T 2ed7d05584d14fdaafb625eaff3ecf08 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:40.913727 9080 raft_consensus.cc:3037] T 2ed7d05584d14fdaafb625eaff3ecf08 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:40.914490 6791 raft_consensus.cc:2454] T 2ed7d05584d14fdaafb625eaff3ecf08 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:40.914584 9080 raft_consensus.cc:2454] T 2ed7d05584d14fdaafb625eaff3ecf08 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:40.914842 8882 leader_election.cc:302] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:40.915053 10731 raft_consensus.cc:2781] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:40.915190 10731 raft_consensus.cc:686] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 13:59:40.915356 10731 consensus_queue.cc:237] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:40.917490 6379 catalog_manager.cc:5484] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:40.935433 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:40.940490 10917 tablet.cc:1722] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:40.941635 10920 tablet.cc:1722] T 905a0cc9091b47029b02bc931c2e0fa7 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:40.944537 10924 tablet.cc:1722] T 905a0cc9091b47029b02bc931c2e0fa7 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:40.949260 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=2ee59d4f31b746c492f6a181301c77b0] alter complete (version 3)
I20240503 13:59:40.952423 10624 raft_consensus.cc:482] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:40.952522 10624 raft_consensus.cc:504] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:40.952963 10624 leader_election.cc:288] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:40.953276 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab63490264c4755aeab897f865e9a11" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:40.953369 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab63490264c4755aeab897f865e9a11" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:40.953415 6922 raft_consensus.cc:2454] T 1ab63490264c4755aeab897f865e9a11 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:40.953501 8947 raft_consensus.cc:2454] T 1ab63490264c4755aeab897f865e9a11 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:40.953732 9016 leader_election.cc:302] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:40.953913 10624 raft_consensus.cc:2781] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:40.953966 10624 raft_consensus.cc:482] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:40.954007 10624 raft_consensus.cc:3037] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:40.954800 10624 raft_consensus.cc:504] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:40.955121 10624 leader_election.cc:288] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:40.955507 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab63490264c4755aeab897f865e9a11" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:40.955508 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab63490264c4755aeab897f865e9a11" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:40.955624 6922 raft_consensus.cc:3037] T 1ab63490264c4755aeab897f865e9a11 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:40.955624 8947 raft_consensus.cc:3037] T 1ab63490264c4755aeab897f865e9a11 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:40.956568 6922 raft_consensus.cc:2454] T 1ab63490264c4755aeab897f865e9a11 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:40.956568 8947 raft_consensus.cc:2454] T 1ab63490264c4755aeab897f865e9a11 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:40.956856 9016 leader_election.cc:302] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:40.957032 10624 raft_consensus.cc:2781] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:40.957129 10624 raft_consensus.cc:686] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 13:59:40.957262 10624 consensus_queue.cc:237] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:40.959604 6379 catalog_manager.cc:5484] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:41.052543 6379 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:41.052706 6379 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:41.055433 6379 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 905a0cc9091b47029b02bc931c2e0fa7
I20240503 13:59:41.056061 6508 tablet_service.cc:1514] Processing DeleteTablet for tablet 905a0cc9091b47029b02bc931c2e0fa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:41 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:41.056062 9190 tablet_service.cc:1514] Processing DeleteTablet for tablet 905a0cc9091b47029b02bc931c2e0fa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:41 UTC) from {username='slave'} at 127.0.0.1:57960
I20240503 13:59:41.056133 9322 tablet_service.cc:1514] Processing DeleteTablet for tablet 905a0cc9091b47029b02bc931c2e0fa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:41 UTC) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:41.056396 10935 tablet_replica.cc:330] T 905a0cc9091b47029b02bc931c2e0fa7 P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:41.056511 10934 tablet_replica.cc:330] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:41.056535 10933 tablet_replica.cc:330] T 905a0cc9091b47029b02bc931c2e0fa7 P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 13:59:41.056614 10935 raft_consensus.cc:2229] T 905a0cc9091b47029b02bc931c2e0fa7 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.056658 10933 raft_consensus.cc:2229] T 905a0cc9091b47029b02bc931c2e0fa7 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.056666 10934 raft_consensus.cc:2229] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:41.056746 10935 raft_consensus.cc:2258] T 905a0cc9091b47029b02bc931c2e0fa7 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.056800 10933 raft_consensus.cc:2258] T 905a0cc9091b47029b02bc931c2e0fa7 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.056871 10934 raft_consensus.cc:2258] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.057307 10935 ts_tablet_manager.cc:1875] T 905a0cc9091b47029b02bc931c2e0fa7 P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:41.057480 10934 ts_tablet_manager.cc:1875] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:41.057739 10933 ts_tablet_manager.cc:1875] T 905a0cc9091b47029b02bc931c2e0fa7 P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:41.059686 10934 ts_tablet_manager.cc:1888] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:41.059769 10935 ts_tablet_manager.cc:1888] T 905a0cc9091b47029b02bc931c2e0fa7 P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:41.059789 10934 log.cc:1196] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/905a0cc9091b47029b02bc931c2e0fa7
I20240503 13:59:41.059875 10935 log.cc:1196] T 905a0cc9091b47029b02bc931c2e0fa7 P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/905a0cc9091b47029b02bc931c2e0fa7
I20240503 13:59:41.060119 10934 ts_tablet_manager.cc:1909] T 905a0cc9091b47029b02bc931c2e0fa7 P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:41.060221 10935 ts_tablet_manager.cc:1909] T 905a0cc9091b47029b02bc931c2e0fa7 P 0d77752181e944698e809e325e5937e6: Deleting consensus metadata
I20240503 13:59:41.060941 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 905a0cc9091b47029b02bc931c2e0fa7 (table rebalancer_dynamic_table_new_name [id=2ee59d4f31b746c492f6a181301c77b0]) successfully deleted
I20240503 13:59:41.061139 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet 905a0cc9091b47029b02bc931c2e0fa7 (table rebalancer_dynamic_table_new_name [id=2ee59d4f31b746c492f6a181301c77b0]) successfully deleted
I20240503 13:59:41.061631 10933 ts_tablet_manager.cc:1888] T 905a0cc9091b47029b02bc931c2e0fa7 P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:41.061702 10933 log.cc:1196] T 905a0cc9091b47029b02bc931c2e0fa7 P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/905a0cc9091b47029b02bc931c2e0fa7
I20240503 13:59:41.061971 10933 ts_tablet_manager.cc:1909] T 905a0cc9091b47029b02bc931c2e0fa7 P ab1d21935d6049888eb0ceccc3c9d735: Deleting consensus metadata
I20240503 13:59:41.062769 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet 905a0cc9091b47029b02bc931c2e0fa7 (table rebalancer_dynamic_table_new_name [id=2ee59d4f31b746c492f6a181301c77b0]) successfully deleted
I20240503 13:59:41.157286 6379 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:41.162338 9061 tablet_service.cc:1467] Processing CreateTablet for tablet 18c7d00b7fb0414d9add38f8ed829756 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0116aa5c611745d1af3a5fc94a060443]), partition=
I20240503 13:59:41.162617 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c7d00b7fb0414d9add38f8ed829756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:41.162624 6771 tablet_service.cc:1467] Processing CreateTablet for tablet 18c7d00b7fb0414d9add38f8ed829756 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0116aa5c611745d1af3a5fc94a060443]), partition=
I20240503 13:59:41.162698 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 18c7d00b7fb0414d9add38f8ed829756 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0116aa5c611745d1af3a5fc94a060443]), partition=
I20240503 13:59:41.162838 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c7d00b7fb0414d9add38f8ed829756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:41.162935 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c7d00b7fb0414d9add38f8ed829756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:41.164994 10908 tablet_bootstrap.cc:492] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:41.165145 10909 tablet_bootstrap.cc:492] T 18c7d00b7fb0414d9add38f8ed829756 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:41.165912 10908 tablet_bootstrap.cc:654] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:41.165957 10909 tablet_bootstrap.cc:654] T 18c7d00b7fb0414d9add38f8ed829756 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:41.166551 10912 tablet_bootstrap.cc:492] T 18c7d00b7fb0414d9add38f8ed829756 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:41.167065 10909 tablet_bootstrap.cc:492] T 18c7d00b7fb0414d9add38f8ed829756 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:41.167146 10909 ts_tablet_manager.cc:1367] T 18c7d00b7fb0414d9add38f8ed829756 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:41.167579 10912 tablet_bootstrap.cc:654] T 18c7d00b7fb0414d9add38f8ed829756 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:41.167774 10909 raft_consensus.cc:348] T 18c7d00b7fb0414d9add38f8ed829756 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:41.167888 10909 raft_consensus.cc:374] T 18c7d00b7fb0414d9add38f8ed829756 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:41.167918 10909 raft_consensus.cc:729] T 18c7d00b7fb0414d9add38f8ed829756 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:41.168048 10909 consensus_queue.cc:260] T 18c7d00b7fb0414d9add38f8ed829756 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:41.168341 10909 ts_tablet_manager.cc:1398] T 18c7d00b7fb0414d9add38f8ed829756 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:41.171586 10912 tablet_bootstrap.cc:492] T 18c7d00b7fb0414d9add38f8ed829756 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:41.171684 10912 ts_tablet_manager.cc:1367] T 18c7d00b7fb0414d9add38f8ed829756 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:41.172171 10912 raft_consensus.cc:348] T 18c7d00b7fb0414d9add38f8ed829756 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:41.172297 10912 raft_consensus.cc:374] T 18c7d00b7fb0414d9add38f8ed829756 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:41.172343 10912 raft_consensus.cc:729] T 18c7d00b7fb0414d9add38f8ed829756 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:41.172469 10912 consensus_queue.cc:260] T 18c7d00b7fb0414d9add38f8ed829756 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:41.172777 10912 ts_tablet_manager.cc:1398] T 18c7d00b7fb0414d9add38f8ed829756 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:41.173767 10908 tablet_bootstrap.cc:492] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:41.173861 10908 ts_tablet_manager.cc:1367] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:59:41.174409 10908 raft_consensus.cc:348] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:41.174520 10908 raft_consensus.cc:374] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:41.174563 10908 raft_consensus.cc:729] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:41.174687 10908 consensus_queue.cc:260] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:41.174989 10908 ts_tablet_manager.cc:1398] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:41.178584 10932 raft_consensus.cc:482] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:41.178673 10932 raft_consensus.cc:504] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:41.179181 10932 leader_election.cc:288] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:41.179494 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c7d00b7fb0414d9add38f8ed829756" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:41.179509 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c7d00b7fb0414d9add38f8ed829756" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:41.179633 8948 raft_consensus.cc:2454] T 18c7d00b7fb0414d9add38f8ed829756 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:41.179653 6791 raft_consensus.cc:2454] T 18c7d00b7fb0414d9add38f8ed829756 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:41.179908 9015 leader_election.cc:302] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:41.180117 10932 raft_consensus.cc:2781] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:41.180166 10932 raft_consensus.cc:482] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:41.180195 10932 raft_consensus.cc:3037] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:41.181221 10932 raft_consensus.cc:504] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:41.181854 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c7d00b7fb0414d9add38f8ed829756" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:41.181970 6791 raft_consensus.cc:3037] T 18c7d00b7fb0414d9add38f8ed829756 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:41.182332 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c7d00b7fb0414d9add38f8ed829756" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:41.182421 8948 raft_consensus.cc:3037] T 18c7d00b7fb0414d9add38f8ed829756 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:41.182453 10932 leader_election.cc:288] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:41.183015 6791 raft_consensus.cc:2454] T 18c7d00b7fb0414d9add38f8ed829756 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:41.183317 9015 leader_election.cc:302] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:41.183487 10932 raft_consensus.cc:2781] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:41.183527 8948 raft_consensus.cc:2454] T 18c7d00b7fb0414d9add38f8ed829756 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:41.183722 10932 raft_consensus.cc:686] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 13:59:41.183869 10932 consensus_queue.cc:237] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:41.185704 6386 catalog_manager.cc:5484] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:41.240690 10851 consensus_queue.cc:1035] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:41.243428 10851 consensus_queue.cc:1035] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:41.289366 6386 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:41.293606 8947 raft_consensus.cc:1261] T 18c7d00b7fb0414d9add38f8ed829756 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:41.293747 6791 raft_consensus.cc:1261] T 18c7d00b7fb0414d9add38f8ed829756 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:41.294036 10930 consensus_queue.cc:1035] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:41.294194 10932 consensus_queue.cc:1035] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:41.296994 10949 tablet.cc:1722] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:41.297417 10949 tablet.cc:1620] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:41.297675 6386 catalog_manager.cc:5576] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365) does not have the latest schema for tablet 18c7d00b7fb0414d9add38f8ed829756 (table rebalancer_dynamic_table [id=0116aa5c611745d1af3a5fc94a060443]). Expected version 1 got 0
I20240503 13:59:41.299268 6379 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet 18c7d00b7fb0414d9add38f8ed829756 (table rebalancer_dynamic_table [id=0116aa5c611745d1af3a5fc94a060443]). Expected version 1 got 0
I20240503 13:59:41.300766 10950 tablet.cc:1722] T 18c7d00b7fb0414d9add38f8ed829756 P 4a85e492c8154e9881a9cc2a105d7e64: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:41.301079 10950 tablet.cc:1620] T 18c7d00b7fb0414d9add38f8ed829756 P 4a85e492c8154e9881a9cc2a105d7e64: MemRowSet was empty: no flush needed.
I20240503 13:59:41.301436 10951 tablet.cc:1722] T 18c7d00b7fb0414d9add38f8ed829756 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:41.301827 10951 tablet.cc:1620] T 18c7d00b7fb0414d9add38f8ed829756 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:41.304306 6386 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=0116aa5c611745d1af3a5fc94a060443] alter complete (version 1)
I20240503 13:59:41.305667 6379 catalog_manager.cc:5576] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365) does not have the latest schema for tablet 18c7d00b7fb0414d9add38f8ed829756 (table rebalancer_dynamic_table [id=0116aa5c611745d1af3a5fc94a060443]). Expected version 1 got 0
I20240503 13:59:41.340211 10936 consensus_queue.cc:1035] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:41.343874 10769 consensus_queue.cc:1035] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:41.358603 10932 consensus_queue.cc:1035] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:41.361709 10930 consensus_queue.cc:1035] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:41.409032 6386 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:41.414294 10949 tablet.cc:1722] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:41.414655 10949 tablet.cc:1620] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:41.415314 10950 tablet.cc:1722] T 18c7d00b7fb0414d9add38f8ed829756 P 4a85e492c8154e9881a9cc2a105d7e64: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:41.415640 10950 tablet.cc:1620] T 18c7d00b7fb0414d9add38f8ed829756 P 4a85e492c8154e9881a9cc2a105d7e64: MemRowSet was empty: no flush needed.
I20240503 13:59:41.416685 10951 tablet.cc:1722] T 18c7d00b7fb0414d9add38f8ed829756 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:41.416993 10951 tablet.cc:1620] T 18c7d00b7fb0414d9add38f8ed829756 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:41.421674 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=0116aa5c611745d1af3a5fc94a060443] alter complete (version 2)
I20240503 13:59:41.489056 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:41.494935 6771 tablet_service.cc:1467] Processing CreateTablet for tablet b5780f4a85b146e2bfbbf7d2feaca316 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=bfd5a93b6dc141158909aa98a0ef7189]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:41.495270 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5780f4a85b146e2bfbbf7d2feaca316. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:41.495309 9322 tablet_service.cc:1467] Processing CreateTablet for tablet db13f399b9e04fe9a69e729e384d046c (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=bfd5a93b6dc141158909aa98a0ef7189]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:41.495568 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db13f399b9e04fe9a69e729e384d046c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:41.495622 9061 tablet_service.cc:1467] Processing CreateTablet for tablet b5780f4a85b146e2bfbbf7d2feaca316 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=bfd5a93b6dc141158909aa98a0ef7189]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:41.495859 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5780f4a85b146e2bfbbf7d2feaca316. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:41.497566 10957 tablet_bootstrap.cc:492] T db13f399b9e04fe9a69e729e384d046c P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:41.497637 10909 tablet_bootstrap.cc:492] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:41.498052 10908 tablet_bootstrap.cc:492] T b5780f4a85b146e2bfbbf7d2feaca316 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:41.498162 9060 tablet_service.cc:1467] Processing CreateTablet for tablet 8f9149a9faa746cd9542aab02a8e2850 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=bfd5a93b6dc141158909aa98a0ef7189]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:41.498379 9060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f9149a9faa746cd9542aab02a8e2850. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:41.498411 6769 tablet_service.cc:1467] Processing CreateTablet for tablet 8f9149a9faa746cd9542aab02a8e2850 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=bfd5a93b6dc141158909aa98a0ef7189]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:41.498620 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f9149a9faa746cd9542aab02a8e2850. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:41.498812 10909 tablet_bootstrap.cc:654] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:41.499034 10908 tablet_bootstrap.cc:654] T b5780f4a85b146e2bfbbf7d2feaca316 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:41.500499 9192 tablet_service.cc:1467] Processing CreateTablet for tablet db13f399b9e04fe9a69e729e384d046c (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=bfd5a93b6dc141158909aa98a0ef7189]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:41.500741 9190 tablet_service.cc:1467] Processing CreateTablet for tablet b5780f4a85b146e2bfbbf7d2feaca316 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=bfd5a93b6dc141158909aa98a0ef7189]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:41.500752 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 8f9149a9faa746cd9542aab02a8e2850 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=bfd5a93b6dc141158909aa98a0ef7189]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:41.500749 9192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db13f399b9e04fe9a69e729e384d046c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:41.501052 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f9149a9faa746cd9542aab02a8e2850. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:41.501317 6771 tablet_service.cc:1467] Processing CreateTablet for tablet db13f399b9e04fe9a69e729e384d046c (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=bfd5a93b6dc141158909aa98a0ef7189]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:41.501504 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db13f399b9e04fe9a69e729e384d046c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:41.502882 10957 tablet_bootstrap.cc:654] T db13f399b9e04fe9a69e729e384d046c P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:41.503609 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5780f4a85b146e2bfbbf7d2feaca316. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:41.504329 10909 tablet_bootstrap.cc:492] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:41.504418 10909 ts_tablet_manager.cc:1367] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:59:41.504506 10957 tablet_bootstrap.cc:492] T db13f399b9e04fe9a69e729e384d046c P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:41.504580 10957 ts_tablet_manager.cc:1367] T db13f399b9e04fe9a69e729e384d046c P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:59:41.504894 10909 raft_consensus.cc:348] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:41.505002 10909 raft_consensus.cc:374] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:41.505033 10909 raft_consensus.cc:729] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:41.504976 10957 raft_consensus.cc:348] T db13f399b9e04fe9a69e729e384d046c P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:41.505074 10957 raft_consensus.cc:374] T db13f399b9e04fe9a69e729e384d046c P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:41.505106 10957 raft_consensus.cc:729] T db13f399b9e04fe9a69e729e384d046c P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:41.505151 10909 consensus_queue.cc:260] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:41.505218 10957 consensus_queue.cc:260] T db13f399b9e04fe9a69e729e384d046c P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:41.505422 10909 ts_tablet_manager.cc:1398] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:41.505488 10957 ts_tablet_manager.cc:1398] T db13f399b9e04fe9a69e729e384d046c P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:41.505604 10909 tablet_bootstrap.cc:492] T 8f9149a9faa746cd9542aab02a8e2850 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:41.505654 10908 tablet_bootstrap.cc:492] T b5780f4a85b146e2bfbbf7d2feaca316 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:41.505744 10908 ts_tablet_manager.cc:1367] T b5780f4a85b146e2bfbbf7d2feaca316 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:59:41.506302 10908 raft_consensus.cc:348] T b5780f4a85b146e2bfbbf7d2feaca316 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:41.506404 10908 raft_consensus.cc:374] T b5780f4a85b146e2bfbbf7d2feaca316 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:41.506438 10908 raft_consensus.cc:729] T b5780f4a85b146e2bfbbf7d2feaca316 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:41.506542 10908 consensus_queue.cc:260] T b5780f4a85b146e2bfbbf7d2feaca316 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:41.506841 10908 ts_tablet_manager.cc:1398] T b5780f4a85b146e2bfbbf7d2feaca316 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:41.506969 10960 tablet_bootstrap.cc:492] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:41.507529 10909 tablet_bootstrap.cc:654] T 8f9149a9faa746cd9542aab02a8e2850 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:41.508155 10960 tablet_bootstrap.cc:654] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:41.509292 10908 tablet_bootstrap.cc:492] T 8f9149a9faa746cd9542aab02a8e2850 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:41.510735 10908 tablet_bootstrap.cc:654] T 8f9149a9faa746cd9542aab02a8e2850 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:41.510815 10909 tablet_bootstrap.cc:492] T 8f9149a9faa746cd9542aab02a8e2850 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:41.510885 10909 ts_tablet_manager.cc:1367] T 8f9149a9faa746cd9542aab02a8e2850 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:41.511265 10909 raft_consensus.cc:348] T 8f9149a9faa746cd9542aab02a8e2850 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:41.511394 10909 raft_consensus.cc:374] T 8f9149a9faa746cd9542aab02a8e2850 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:41.511430 10909 raft_consensus.cc:729] T 8f9149a9faa746cd9542aab02a8e2850 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:41.511971 10960 tablet_bootstrap.cc:492] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:41.512053 10960 ts_tablet_manager.cc:1367] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:41.512050 10908 tablet_bootstrap.cc:492] T 8f9149a9faa746cd9542aab02a8e2850 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:41.512147 10908 ts_tablet_manager.cc:1367] T 8f9149a9faa746cd9542aab02a8e2850 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:41.512518 10960 raft_consensus.cc:348] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:41.512627 10960 raft_consensus.cc:374] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:41.512661 10960 raft_consensus.cc:729] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:41.512622 10908 raft_consensus.cc:348] T 8f9149a9faa746cd9542aab02a8e2850 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:41.512717 10908 raft_consensus.cc:374] T 8f9149a9faa746cd9542aab02a8e2850 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:41.512755 10908 raft_consensus.cc:729] T 8f9149a9faa746cd9542aab02a8e2850 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:41.512779 10960 consensus_queue.cc:260] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:41.512797 10909 consensus_queue.cc:260] T 8f9149a9faa746cd9542aab02a8e2850 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:41.512856 10908 consensus_queue.cc:260] T 8f9149a9faa746cd9542aab02a8e2850 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:41.513070 10960 ts_tablet_manager.cc:1398] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:41.513108 10908 ts_tablet_manager.cc:1398] T 8f9149a9faa746cd9542aab02a8e2850 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:41.513343 10960 tablet_bootstrap.cc:492] T b5780f4a85b146e2bfbbf7d2feaca316 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:41.514173 10960 tablet_bootstrap.cc:654] T b5780f4a85b146e2bfbbf7d2feaca316 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:41.514133 10909 ts_tablet_manager.cc:1398] T 8f9149a9faa746cd9542aab02a8e2850 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:59:41.515201 10909 tablet_bootstrap.cc:492] T db13f399b9e04fe9a69e729e384d046c P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:41.516177 10909 tablet_bootstrap.cc:654] T db13f399b9e04fe9a69e729e384d046c P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:41.516181 10960 tablet_bootstrap.cc:492] T b5780f4a85b146e2bfbbf7d2feaca316 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:41.516259 10960 ts_tablet_manager.cc:1367] T b5780f4a85b146e2bfbbf7d2feaca316 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:41.516721 10960 raft_consensus.cc:348] T b5780f4a85b146e2bfbbf7d2feaca316 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:41.516834 10960 raft_consensus.cc:374] T b5780f4a85b146e2bfbbf7d2feaca316 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:41.516880 10960 raft_consensus.cc:729] T b5780f4a85b146e2bfbbf7d2feaca316 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:41.517007 10960 consensus_queue.cc:260] T b5780f4a85b146e2bfbbf7d2feaca316 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:41.517275 10960 ts_tablet_manager.cc:1398] T b5780f4a85b146e2bfbbf7d2feaca316 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:41.517597 10961 tablet_bootstrap.cc:492] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:41.518647 10961 tablet_bootstrap.cc:654] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:41.519155 10909 tablet_bootstrap.cc:492] T db13f399b9e04fe9a69e729e384d046c P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:41.519241 10909 ts_tablet_manager.cc:1367] T db13f399b9e04fe9a69e729e384d046c P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 13:59:41.519661 10909 raft_consensus.cc:348] T db13f399b9e04fe9a69e729e384d046c P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:41.519773 10909 raft_consensus.cc:374] T db13f399b9e04fe9a69e729e384d046c P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:41.519814 10961 tablet_bootstrap.cc:492] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:41.519829 10909 raft_consensus.cc:729] T db13f399b9e04fe9a69e729e384d046c P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:41.519889 10961 ts_tablet_manager.cc:1367] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:41.519956 10909 consensus_queue.cc:260] T db13f399b9e04fe9a69e729e384d046c P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:41.520203 10909 ts_tablet_manager.cc:1398] T db13f399b9e04fe9a69e729e384d046c P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:41.520291 10961 raft_consensus.cc:348] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:41.520406 10961 raft_consensus.cc:374] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:41.520437 10961 raft_consensus.cc:729] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:41.520618 10961 consensus_queue.cc:260] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:41.520959 10961 ts_tablet_manager.cc:1398] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:41.524801 6386 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:41.529886 10949 tablet.cc:1722] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:41.530385 10950 tablet.cc:1722] T 18c7d00b7fb0414d9add38f8ed829756 P 4a85e492c8154e9881a9cc2a105d7e64: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:41.530428 10951 tablet.cc:1722] T 18c7d00b7fb0414d9add38f8ed829756 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:41.535306 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=0116aa5c611745d1af3a5fc94a060443] alter complete (version 3)
I20240503 13:59:41.601732 10851 raft_consensus.cc:482] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:41.601863 10851 raft_consensus.cc:504] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:41.602366 10851 leader_election.cc:288] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:41.602670 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db13f399b9e04fe9a69e729e384d046c" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:41.602712 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db13f399b9e04fe9a69e729e384d046c" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:41.602808 9343 raft_consensus.cc:2454] T db13f399b9e04fe9a69e729e384d046c P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:41.602855 6791 raft_consensus.cc:2454] T db13f399b9e04fe9a69e729e384d046c P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:41.603085 9148 leader_election.cc:302] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:41.603257 10851 raft_consensus.cc:2781] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:41.603348 10851 raft_consensus.cc:482] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:41.603392 10851 raft_consensus.cc:3037] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:41.604259 10851 raft_consensus.cc:504] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:41.604533 10851 leader_election.cc:288] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:41.604812 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db13f399b9e04fe9a69e729e384d046c" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:41.604861 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db13f399b9e04fe9a69e729e384d046c" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:41.604907 9343 raft_consensus.cc:3037] T db13f399b9e04fe9a69e729e384d046c P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:41.604997 6791 raft_consensus.cc:3037] T db13f399b9e04fe9a69e729e384d046c P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:41.605757 6791 raft_consensus.cc:2454] T db13f399b9e04fe9a69e729e384d046c P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:41.605779 9343 raft_consensus.cc:2454] T db13f399b9e04fe9a69e729e384d046c P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:41.606045 9148 leader_election.cc:302] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:41.606243 10851 raft_consensus.cc:2781] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:41.606333 10851 raft_consensus.cc:686] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 13:59:41.606487 10851 consensus_queue.cc:237] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:41.608736 6379 catalog_manager.cc:5484] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:41.639259 6379 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:41.639542 6379 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:41.642189 6379 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 18c7d00b7fb0414d9add38f8ed829756
I20240503 13:59:41.642932 8928 tablet_service.cc:1514] Processing DeleteTablet for tablet 18c7d00b7fb0414d9add38f8ed829756 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:41 UTC) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:41.642944 6771 tablet_service.cc:1514] Processing DeleteTablet for tablet 18c7d00b7fb0414d9add38f8ed829756 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:41 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:41.643268 10971 tablet_replica.cc:330] T 18c7d00b7fb0414d9add38f8ed829756 P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:41.643385 9060 tablet_service.cc:1514] Processing DeleteTablet for tablet 18c7d00b7fb0414d9add38f8ed829756 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:41 UTC) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:41.643487 10971 raft_consensus.cc:2229] T 18c7d00b7fb0414d9add38f8ed829756 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.643635 10971 raft_consensus.cc:2258] T 18c7d00b7fb0414d9add38f8ed829756 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.643708 10972 tablet_replica.cc:330] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:41.644068 10972 raft_consensus.cc:2229] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:41.644254 10971 ts_tablet_manager.cc:1875] T 18c7d00b7fb0414d9add38f8ed829756 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:41.644304 10972 raft_consensus.cc:2258] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.644830 10972 ts_tablet_manager.cc:1875] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:41.646741 10971 ts_tablet_manager.cc:1888] T 18c7d00b7fb0414d9add38f8ed829756 P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:41.646876 10971 log.cc:1196] T 18c7d00b7fb0414d9add38f8ed829756 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/18c7d00b7fb0414d9add38f8ed829756
I20240503 13:59:41.647238 10971 ts_tablet_manager.cc:1909] T 18c7d00b7fb0414d9add38f8ed829756 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:41.647434 10972 ts_tablet_manager.cc:1888] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:41.647508 10972 log.cc:1196] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/18c7d00b7fb0414d9add38f8ed829756
I20240503 13:59:41.647943 10972 ts_tablet_manager.cc:1909] T 18c7d00b7fb0414d9add38f8ed829756 P 8d574fb4a89d4eea947c859470d44dd4: Deleting consensus metadata
I20240503 13:59:41.647938 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet 18c7d00b7fb0414d9add38f8ed829756 (table rebalancer_dynamic_table_new_name [id=0116aa5c611745d1af3a5fc94a060443]) successfully deleted
I20240503 13:59:41.648928 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet 18c7d00b7fb0414d9add38f8ed829756 (table rebalancer_dynamic_table_new_name [id=0116aa5c611745d1af3a5fc94a060443]) successfully deleted
I20240503 13:59:41.650524 10737 raft_consensus.cc:482] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:41.650622 10737 raft_consensus.cc:504] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:41.651139 10737 leader_election.cc:288] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:41.651204 10970 tablet_replica.cc:330] T 18c7d00b7fb0414d9add38f8ed829756 P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:41.651372 10970 raft_consensus.cc:2229] T 18c7d00b7fb0414d9add38f8ed829756 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.651501 10970 raft_consensus.cc:2258] T 18c7d00b7fb0414d9add38f8ed829756 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.651639 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5780f4a85b146e2bfbbf7d2feaca316" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:41.651762 9081 raft_consensus.cc:2454] T b5780f4a85b146e2bfbbf7d2feaca316 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:41.652065 6726 leader_election.cc:302] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:41.652170 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5780f4a85b146e2bfbbf7d2feaca316" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:41.652230 10737 raft_consensus.cc:2781] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:41.652276 10737 raft_consensus.cc:482] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:41.652315 10737 raft_consensus.cc:3037] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:41.652308 9205 raft_consensus.cc:2454] T b5780f4a85b146e2bfbbf7d2feaca316 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:41.652330 10970 ts_tablet_manager.cc:1875] T 18c7d00b7fb0414d9add38f8ed829756 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:41.653254 10737 raft_consensus.cc:504] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:41.653648 10737 leader_election.cc:288] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:41.653952 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5780f4a85b146e2bfbbf7d2feaca316" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:41.653980 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5780f4a85b146e2bfbbf7d2feaca316" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:41.654067 9205 raft_consensus.cc:3037] T b5780f4a85b146e2bfbbf7d2feaca316 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:41.654083 9081 raft_consensus.cc:3037] T b5780f4a85b146e2bfbbf7d2feaca316 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:41.654618 10970 ts_tablet_manager.cc:1888] T 18c7d00b7fb0414d9add38f8ed829756 P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:41.654685 10970 log.cc:1196] T 18c7d00b7fb0414d9add38f8ed829756 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/18c7d00b7fb0414d9add38f8ed829756
I20240503 13:59:41.655004 10970 ts_tablet_manager.cc:1909] T 18c7d00b7fb0414d9add38f8ed829756 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting consensus metadata
I20240503 13:59:41.655004 9081 raft_consensus.cc:2454] T b5780f4a85b146e2bfbbf7d2feaca316 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:41.655025 9205 raft_consensus.cc:2454] T b5780f4a85b146e2bfbbf7d2feaca316 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:41.655253 6726 leader_election.cc:302] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:41.655429 10737 raft_consensus.cc:2781] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:41.655524 10737 raft_consensus.cc:686] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:41.655656 10737 consensus_queue.cc:237] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:41.655911 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 18c7d00b7fb0414d9add38f8ed829756 (table rebalancer_dynamic_table_new_name [id=0116aa5c611745d1af3a5fc94a060443]) successfully deleted
I20240503 13:59:41.657830 6379 catalog_manager.cc:5484] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:41.668543 10928 raft_consensus.cc:482] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:41.668646 10928 raft_consensus.cc:504] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:41.669137 10928 leader_election.cc:288] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:41.669481 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f9149a9faa746cd9542aab02a8e2850" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:41.669493 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f9149a9faa746cd9542aab02a8e2850" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:41.669615 6791 raft_consensus.cc:2454] T 8f9149a9faa746cd9542aab02a8e2850 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:41.669620 9081 raft_consensus.cc:2454] T 8f9149a9faa746cd9542aab02a8e2850 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:41.669888 6595 leader_election.cc:302] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:41.670092 10928 raft_consensus.cc:2781] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:41.670163 10928 raft_consensus.cc:482] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:41.670213 10928 raft_consensus.cc:3037] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:41.671162 10928 raft_consensus.cc:504] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:41.671489 10928 leader_election.cc:288] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:41.671782 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f9149a9faa746cd9542aab02a8e2850" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:41.671913 9081 raft_consensus.cc:3037] T 8f9149a9faa746cd9542aab02a8e2850 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:41.672839 9081 raft_consensus.cc:2454] T 8f9149a9faa746cd9542aab02a8e2850 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:41.673229 6595 leader_election.cc:302] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:41.673262 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f9149a9faa746cd9542aab02a8e2850" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:41.673362 6791 raft_consensus.cc:3037] T 8f9149a9faa746cd9542aab02a8e2850 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:41.673454 10928 raft_consensus.cc:2781] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:41.673568 10928 raft_consensus.cc:686] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 13:59:41.673697 10928 consensus_queue.cc:237] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:41.674387 6791 raft_consensus.cc:2454] T 8f9149a9faa746cd9542aab02a8e2850 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:41.676036 6379 catalog_manager.cc:5484] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:41.743763 6379 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:41.748953 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 01f42d12a5074af9bbe991853990684b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f29a2b35a7af48659aea91b865e44a33]), partition=
I20240503 13:59:41.749197 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01f42d12a5074af9bbe991853990684b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:41.749241 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 01f42d12a5074af9bbe991853990684b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f29a2b35a7af48659aea91b865e44a33]), partition=
I20240503 13:59:41.749477 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01f42d12a5074af9bbe991853990684b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:41.751504 10975 tablet_bootstrap.cc:492] T 01f42d12a5074af9bbe991853990684b P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:41.751534 9322 tablet_service.cc:1467] Processing CreateTablet for tablet 01f42d12a5074af9bbe991853990684b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f29a2b35a7af48659aea91b865e44a33]), partition=
I20240503 13:59:41.751742 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01f42d12a5074af9bbe991853990684b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:41.752523 10975 tablet_bootstrap.cc:654] T 01f42d12a5074af9bbe991853990684b P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:41.753598 10957 tablet_bootstrap.cc:492] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:41.754281 10961 tablet_bootstrap.cc:492] T 01f42d12a5074af9bbe991853990684b P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:41.754643 10957 tablet_bootstrap.cc:654] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:41.755200 10961 tablet_bootstrap.cc:654] T 01f42d12a5074af9bbe991853990684b P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:41.756141 10957 tablet_bootstrap.cc:492] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:41.756234 10957 ts_tablet_manager.cc:1367] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 13:59:41.756249 10961 tablet_bootstrap.cc:492] T 01f42d12a5074af9bbe991853990684b P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:41.756328 10961 ts_tablet_manager.cc:1367] T 01f42d12a5074af9bbe991853990684b P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:41.756631 10957 raft_consensus.cc:348] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:41.756727 10957 raft_consensus.cc:374] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:41.756759 10957 raft_consensus.cc:729] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:41.756729 10961 raft_consensus.cc:348] T 01f42d12a5074af9bbe991853990684b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:41.756858 10961 raft_consensus.cc:374] T 01f42d12a5074af9bbe991853990684b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:41.756901 10961 raft_consensus.cc:729] T 01f42d12a5074af9bbe991853990684b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:41.756927 10957 consensus_queue.cc:260] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:41.757019 10961 consensus_queue.cc:260] T 01f42d12a5074af9bbe991853990684b P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:41.757243 10957 ts_tablet_manager.cc:1398] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:41.757302 10961 ts_tablet_manager.cc:1398] T 01f42d12a5074af9bbe991853990684b P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:41.760241 10975 tablet_bootstrap.cc:492] T 01f42d12a5074af9bbe991853990684b P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:41.760540 10975 ts_tablet_manager.cc:1367] T 01f42d12a5074af9bbe991853990684b P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:59:41.761128 10975 raft_consensus.cc:348] T 01f42d12a5074af9bbe991853990684b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:41.761441 10975 raft_consensus.cc:374] T 01f42d12a5074af9bbe991853990684b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:41.761533 10975 raft_consensus.cc:729] T 01f42d12a5074af9bbe991853990684b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:41.761788 10975 consensus_queue.cc:260] T 01f42d12a5074af9bbe991853990684b P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:41.762230 10975 ts_tablet_manager.cc:1398] T 01f42d12a5074af9bbe991853990684b P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:41.838452 10751 raft_consensus.cc:482] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:41.838589 10751 raft_consensus.cc:504] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:41.839012 10751 leader_election.cc:288] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:41.839534 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f42d12a5074af9bbe991853990684b" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:41.839691 6658 raft_consensus.cc:2454] T 01f42d12a5074af9bbe991853990684b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:41.839977 9279 leader_election.cc:302] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 2145d15ab2d74c0287839266f1411a43; no voters:
I20240503 13:59:41.840178 10751 raft_consensus.cc:2781] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:41.840250 10751 raft_consensus.cc:482] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:41.840198 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f42d12a5074af9bbe991853990684b" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:41.840286 10751 raft_consensus.cc:3037] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:41.840337 8947 raft_consensus.cc:2454] T 01f42d12a5074af9bbe991853990684b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:41.841138 10751 raft_consensus.cc:504] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:41.841820 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f42d12a5074af9bbe991853990684b" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:41.841923 6658 raft_consensus.cc:3037] T 01f42d12a5074af9bbe991853990684b P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:41.842023 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f42d12a5074af9bbe991853990684b" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:41.842123 8947 raft_consensus.cc:3037] T 01f42d12a5074af9bbe991853990684b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:41.843024 6658 raft_consensus.cc:2454] T 01f42d12a5074af9bbe991853990684b P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:41.843174 8947 raft_consensus.cc:2454] T 01f42d12a5074af9bbe991853990684b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:41.843189 10751 leader_election.cc:288] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:41.843475 9278 leader_election.cc:302] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 0d77752181e944698e809e325e5937e6; no voters:
I20240503 13:59:41.843703 10751 raft_consensus.cc:2781] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:41.843796 10751 raft_consensus.cc:686] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:41.843942 10751 consensus_queue.cc:237] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:41.846170 6379 catalog_manager.cc:5484] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:41.971364 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:41.975170 6658 raft_consensus.cc:1261] T 01f42d12a5074af9bbe991853990684b P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:41.975270 8947 raft_consensus.cc:1261] T 01f42d12a5074af9bbe991853990684b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:41.975708 10564 consensus_queue.cc:1035] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:41.976123 10564 consensus_queue.cc:1035] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:41.978883 6379 catalog_manager.cc:5576] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119) does not have the latest schema for tablet 01f42d12a5074af9bbe991853990684b (table rebalancer_dynamic_table [id=f29a2b35a7af48659aea91b865e44a33]). Expected version 1 got 0
I20240503 13:59:41.979727 6386 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet 01f42d12a5074af9bbe991853990684b (table rebalancer_dynamic_table [id=f29a2b35a7af48659aea91b865e44a33]). Expected version 1 got 0
I20240503 13:59:41.981142 10988 tablet.cc:1722] T 01f42d12a5074af9bbe991853990684b P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:41.981547 10988 tablet.cc:1620] T 01f42d12a5074af9bbe991853990684b P 2145d15ab2d74c0287839266f1411a43: MemRowSet was empty: no flush needed.
I20240503 13:59:41.981851 10951 tablet.cc:1722] T 01f42d12a5074af9bbe991853990684b P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:41.982156 10951 tablet.cc:1620] T 01f42d12a5074af9bbe991853990684b P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:41.982635 6386 catalog_manager.cc:5576] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931) does not have the latest schema for tablet 01f42d12a5074af9bbe991853990684b (table rebalancer_dynamic_table [id=f29a2b35a7af48659aea91b865e44a33]). Expected version 1 got 0
I20240503 13:59:41.985283 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=f29a2b35a7af48659aea91b865e44a33] alter complete (version 1)
I20240503 13:59:41.986869 6379 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet 01f42d12a5074af9bbe991853990684b (table rebalancer_dynamic_table [id=f29a2b35a7af48659aea91b865e44a33]). Expected version 1 got 0
I20240503 13:59:41.987008 10987 tablet.cc:1722] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:41.987367 10987 tablet.cc:1620] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:42.017164 10851 consensus_queue.cc:1035] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:42.020260 10851 consensus_queue.cc:1035] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:42.088591 6386 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:42.093832 10987 tablet.cc:1722] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:42.094182 10987 tablet.cc:1620] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:42.094290 10951 tablet.cc:1722] T 01f42d12a5074af9bbe991853990684b P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:42.094596 10951 tablet.cc:1620] T 01f42d12a5074af9bbe991853990684b P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:42.099599 10988 tablet.cc:1722] T 01f42d12a5074af9bbe991853990684b P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:42.100174 10988 tablet.cc:1620] T 01f42d12a5074af9bbe991853990684b P 2145d15ab2d74c0287839266f1411a43: MemRowSet was empty: no flush needed.
I20240503 13:59:42.100360 6386 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=f29a2b35a7af48659aea91b865e44a33] alter complete (version 2)
I20240503 13:59:42.106534 10737 consensus_queue.cc:1035] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:42.111380 10906 consensus_queue.cc:1035] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:42.189615 10928 consensus_queue.cc:1035] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:42.193153 10996 consensus_queue.cc:1035] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:42.199447 6386 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:42.204401 6386 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:42.206478 9322 tablet_service.cc:1467] Processing CreateTablet for tablet f3dd70554b644cddb1e8eefe7e4ea646 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=2f45bc2e41c6493a85eab02853317fd7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:42.206584 6640 tablet_service.cc:1467] Processing CreateTablet for tablet f3dd70554b644cddb1e8eefe7e4ea646 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=2f45bc2e41c6493a85eab02853317fd7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:42.206804 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3dd70554b644cddb1e8eefe7e4ea646. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:42.206894 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3dd70554b644cddb1e8eefe7e4ea646. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:42.207125 9321 tablet_service.cc:1467] Processing CreateTablet for tablet 47d38e1d60dd42eabf7191598ded8cd5 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=2f45bc2e41c6493a85eab02853317fd7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:42.207551 9321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d38e1d60dd42eabf7191598ded8cd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:42.209282 10961 tablet_bootstrap.cc:492] T f3dd70554b644cddb1e8eefe7e4ea646 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:42.209473 10957 tablet_bootstrap.cc:492] T f3dd70554b644cddb1e8eefe7e4ea646 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:42.210407 10957 tablet_bootstrap.cc:654] T f3dd70554b644cddb1e8eefe7e4ea646 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:42.210448 9321 tablet_service.cc:1467] Processing CreateTablet for tablet 1350225b1a2848debca68e1bad5528c5 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=2f45bc2e41c6493a85eab02853317fd7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:42.210641 9321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1350225b1a2848debca68e1bad5528c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:42.210749 10961 tablet_bootstrap.cc:654] T f3dd70554b644cddb1e8eefe7e4ea646 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:42.211884 6507 tablet_service.cc:1467] Processing CreateTablet for tablet 1350225b1a2848debca68e1bad5528c5 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=2f45bc2e41c6493a85eab02853317fd7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:42.211884 6508 tablet_service.cc:1467] Processing CreateTablet for tablet f3dd70554b644cddb1e8eefe7e4ea646 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=2f45bc2e41c6493a85eab02853317fd7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:42.212189 6507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1350225b1a2848debca68e1bad5528c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:42.212319 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3dd70554b644cddb1e8eefe7e4ea646. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:42.212401 9190 tablet_service.cc:1467] Processing CreateTablet for tablet 47d38e1d60dd42eabf7191598ded8cd5 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=2f45bc2e41c6493a85eab02853317fd7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:42.212662 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d38e1d60dd42eabf7191598ded8cd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:42.212654 10957 tablet_bootstrap.cc:492] T f3dd70554b644cddb1e8eefe7e4ea646 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:42.212743 10957 ts_tablet_manager.cc:1367] T f3dd70554b644cddb1e8eefe7e4ea646 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:59:42.213228 10957 raft_consensus.cc:348] T f3dd70554b644cddb1e8eefe7e4ea646 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:42.213361 10957 raft_consensus.cc:374] T f3dd70554b644cddb1e8eefe7e4ea646 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:42.213399 10957 raft_consensus.cc:729] T f3dd70554b644cddb1e8eefe7e4ea646 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:42.213536 10957 consensus_queue.cc:260] T f3dd70554b644cddb1e8eefe7e4ea646 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:42.213821 10957 ts_tablet_manager.cc:1398] T f3dd70554b644cddb1e8eefe7e4ea646 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:42.214825 10957 tablet_bootstrap.cc:492] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:42.215644 10961 tablet_bootstrap.cc:492] T f3dd70554b644cddb1e8eefe7e4ea646 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:42.215752 10961 ts_tablet_manager.cc:1367] T f3dd70554b644cddb1e8eefe7e4ea646 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 13:59:42.216210 10961 raft_consensus.cc:348] T f3dd70554b644cddb1e8eefe7e4ea646 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:42.216313 10961 raft_consensus.cc:374] T f3dd70554b644cddb1e8eefe7e4ea646 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:42.216348 10961 raft_consensus.cc:729] T f3dd70554b644cddb1e8eefe7e4ea646 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:42.216322 6901 tablet_service.cc:1467] Processing CreateTablet for tablet 1350225b1a2848debca68e1bad5528c5 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=2f45bc2e41c6493a85eab02853317fd7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:42.216466 10961 consensus_queue.cc:260] T f3dd70554b644cddb1e8eefe7e4ea646 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:42.216619 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1350225b1a2848debca68e1bad5528c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:42.216753 10961 ts_tablet_manager.cc:1398] T f3dd70554b644cddb1e8eefe7e4ea646 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:42.216322 6900 tablet_service.cc:1467] Processing CreateTablet for tablet 47d38e1d60dd42eabf7191598ded8cd5 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=2f45bc2e41c6493a85eab02853317fd7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:42.216961 6900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d38e1d60dd42eabf7191598ded8cd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:42.217061 11000 tablet_bootstrap.cc:492] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:42.218483 11000 tablet_bootstrap.cc:654] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:42.218864 10957 tablet_bootstrap.cc:654] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:42.221249 10987 tablet.cc:1722] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:42.221823 10957 tablet_bootstrap.cc:492] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:42.221983 10957 ts_tablet_manager.cc:1367] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 13:59:42.222748 10951 tablet.cc:1722] T 01f42d12a5074af9bbe991853990684b P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:42.222883 10988 tablet.cc:1722] T 01f42d12a5074af9bbe991853990684b P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:42.227169 11001 tablet_bootstrap.cc:492] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:42.228170 11001 tablet_bootstrap.cc:654] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:42.228256 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=f29a2b35a7af48659aea91b865e44a33] alter complete (version 3)
I20240503 13:59:42.228880 11002 tablet_bootstrap.cc:492] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:42.229846 11002 tablet_bootstrap.cc:654] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:42.230825 11001 tablet_bootstrap.cc:492] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:42.231343 11002 tablet_bootstrap.cc:492] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:42.231485 11001 ts_tablet_manager.cc:1367] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:42.231626 11002 ts_tablet_manager.cc:1367] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:42.232033 11002 raft_consensus.cc:348] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:42.232134 11002 raft_consensus.cc:374] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:42.232609 11001 raft_consensus.cc:348] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:42.232700 11001 raft_consensus.cc:374] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:42.232734 11001 raft_consensus.cc:729] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:42.232868 11001 consensus_queue.cc:260] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:42.233137 11001 ts_tablet_manager.cc:1398] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:42.232168 11002 raft_consensus.cc:729] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:42.233472 11002 consensus_queue.cc:260] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:42.233855 11002 ts_tablet_manager.cc:1398] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:42.234215 11002 tablet_bootstrap.cc:492] T 47d38e1d60dd42eabf7191598ded8cd5 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:42.235205 11002 tablet_bootstrap.cc:654] T 47d38e1d60dd42eabf7191598ded8cd5 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:42.235546 10957 raft_consensus.cc:348] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:42.236428 10957 raft_consensus.cc:374] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:42.237115 10957 raft_consensus.cc:729] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:42.237553 10957 consensus_queue.cc:260] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:42.237723 11002 tablet_bootstrap.cc:492] T 47d38e1d60dd42eabf7191598ded8cd5 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:42.237816 11002 ts_tablet_manager.cc:1367] T 47d38e1d60dd42eabf7191598ded8cd5 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:42.238449 11002 raft_consensus.cc:348] T 47d38e1d60dd42eabf7191598ded8cd5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:42.238564 11002 raft_consensus.cc:374] T 47d38e1d60dd42eabf7191598ded8cd5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:42.238631 11002 raft_consensus.cc:729] T 47d38e1d60dd42eabf7191598ded8cd5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:42.238952 11002 consensus_queue.cc:260] T 47d38e1d60dd42eabf7191598ded8cd5 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:42.239385 11002 ts_tablet_manager.cc:1398] T 47d38e1d60dd42eabf7191598ded8cd5 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:42.240803 10957 ts_tablet_manager.cc:1398] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.018s user 0.000s sys 0.001s
I20240503 13:59:42.241420 10957 tablet_bootstrap.cc:492] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:42.242518 10957 tablet_bootstrap.cc:654] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:42.244513 10957 tablet_bootstrap.cc:492] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:42.244596 10957 ts_tablet_manager.cc:1367] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:59:42.245016 10957 raft_consensus.cc:348] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:42.245141 10957 raft_consensus.cc:374] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:42.245275 10957 raft_consensus.cc:729] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:42.245414 10957 consensus_queue.cc:260] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:42.245973 10957 ts_tablet_manager.cc:1398] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:42.254415 11000 tablet_bootstrap.cc:492] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:42.254540 11000 ts_tablet_manager.cc:1367] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.000s
I20240503 13:59:42.255103 11000 raft_consensus.cc:348] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:42.255221 11000 raft_consensus.cc:374] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:42.255254 11000 raft_consensus.cc:729] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:42.255636 11000 consensus_queue.cc:260] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:42.255964 11000 ts_tablet_manager.cc:1398] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:42.256136 11000 tablet_bootstrap.cc:492] T 1350225b1a2848debca68e1bad5528c5 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:42.257194 11000 tablet_bootstrap.cc:654] T 1350225b1a2848debca68e1bad5528c5 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:42.258251 11000 tablet_bootstrap.cc:492] T 1350225b1a2848debca68e1bad5528c5 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:42.258328 11000 ts_tablet_manager.cc:1367] T 1350225b1a2848debca68e1bad5528c5 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:42.259696 11000 raft_consensus.cc:348] T 1350225b1a2848debca68e1bad5528c5 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:42.259827 11000 raft_consensus.cc:374] T 1350225b1a2848debca68e1bad5528c5 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:42.259876 11000 raft_consensus.cc:729] T 1350225b1a2848debca68e1bad5528c5 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:42.260035 11000 consensus_queue.cc:260] T 1350225b1a2848debca68e1bad5528c5 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:42.260443 11000 ts_tablet_manager.cc:1398] T 1350225b1a2848debca68e1bad5528c5 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:42.283597 10751 raft_consensus.cc:482] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:42.283710 10751 raft_consensus.cc:504] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:42.284085 10751 leader_election.cc:288] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:42.284548 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1350225b1a2848debca68e1bad5528c5" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:42.284559 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1350225b1a2848debca68e1bad5528c5" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:42.284682 6528 raft_consensus.cc:2454] T 1350225b1a2848debca68e1bad5528c5 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:42.284682 6922 raft_consensus.cc:2454] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:42.285009 9278 leader_election.cc:302] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:42.285205 10751 raft_consensus.cc:2781] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:42.285264 10751 raft_consensus.cc:482] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:42.285291 10751 raft_consensus.cc:3037] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:42.286139 10751 raft_consensus.cc:504] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:42.286432 10751 leader_election.cc:288] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:42.286737 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1350225b1a2848debca68e1bad5528c5" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:42.286803 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1350225b1a2848debca68e1bad5528c5" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:42.286852 6528 raft_consensus.cc:3037] T 1350225b1a2848debca68e1bad5528c5 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:42.286950 6922 raft_consensus.cc:3037] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:42.287789 6922 raft_consensus.cc:2454] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:42.287902 6528 raft_consensus.cc:2454] T 1350225b1a2848debca68e1bad5528c5 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:42.288059 9278 leader_election.cc:302] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:42.288347 10751 raft_consensus.cc:2781] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:42.288439 10751 raft_consensus.cc:686] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:42.288578 10751 consensus_queue.cc:237] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:42.290894 6379 catalog_manager.cc:5484] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:42.332038 6386 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:42.332201 6386 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:42.334262 6386 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 01f42d12a5074af9bbe991853990684b
I20240503 13:59:42.334816 6640 tablet_service.cc:1514] Processing DeleteTablet for tablet 01f42d12a5074af9bbe991853990684b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:42 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:42.334818 8928 tablet_service.cc:1514] Processing DeleteTablet for tablet 01f42d12a5074af9bbe991853990684b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:42 UTC) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:42.334934 9321 tablet_service.cc:1514] Processing DeleteTablet for tablet 01f42d12a5074af9bbe991853990684b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:42 UTC) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:42.335163 11012 tablet_replica.cc:330] T 01f42d12a5074af9bbe991853990684b P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:42.335363 11013 tablet_replica.cc:330] T 01f42d12a5074af9bbe991853990684b P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:42.335426 11012 raft_consensus.cc:2229] T 01f42d12a5074af9bbe991853990684b P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:42.335578 11012 raft_consensus.cc:2258] T 01f42d12a5074af9bbe991853990684b P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:42.335645 11014 tablet_replica.cc:330] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:42.335811 11014 raft_consensus.cc:2229] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:42.336055 11014 raft_consensus.cc:2258] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:42.336298 11012 ts_tablet_manager.cc:1875] T 01f42d12a5074af9bbe991853990684b P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:42.336680 11014 ts_tablet_manager.cc:1875] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:42.336843 11013 raft_consensus.cc:2229] T 01f42d12a5074af9bbe991853990684b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:42.337034 11013 raft_consensus.cc:2258] T 01f42d12a5074af9bbe991853990684b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:42.337760 11013 ts_tablet_manager.cc:1875] T 01f42d12a5074af9bbe991853990684b P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:42.338788 11014 ts_tablet_manager.cc:1888] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:42.338891 11014 log.cc:1196] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/01f42d12a5074af9bbe991853990684b
I20240503 13:59:42.339212 11014 ts_tablet_manager.cc:1909] T 01f42d12a5074af9bbe991853990684b P 0d77752181e944698e809e325e5937e6: Deleting consensus metadata
I20240503 13:59:42.339628 11012 ts_tablet_manager.cc:1888] T 01f42d12a5074af9bbe991853990684b P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:42.339732 11012 log.cc:1196] T 01f42d12a5074af9bbe991853990684b P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/01f42d12a5074af9bbe991853990684b
I20240503 13:59:42.339972 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet 01f42d12a5074af9bbe991853990684b (table rebalancer_dynamic_table_new_name [id=f29a2b35a7af48659aea91b865e44a33]) successfully deleted
I20240503 13:59:42.340164 11012 ts_tablet_manager.cc:1909] T 01f42d12a5074af9bbe991853990684b P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:42.340440 11013 ts_tablet_manager.cc:1888] T 01f42d12a5074af9bbe991853990684b P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:42.340520 11013 log.cc:1196] T 01f42d12a5074af9bbe991853990684b P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/01f42d12a5074af9bbe991853990684b
I20240503 13:59:42.340831 11013 ts_tablet_manager.cc:1909] T 01f42d12a5074af9bbe991853990684b P 06ca8dbc004c42b99faa527a4e6f9545: Deleting consensus metadata
I20240503 13:59:42.340870 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet 01f42d12a5074af9bbe991853990684b (table rebalancer_dynamic_table_new_name [id=f29a2b35a7af48659aea91b865e44a33]) successfully deleted
I20240503 13:59:42.341480 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 01f42d12a5074af9bbe991853990684b (table rebalancer_dynamic_table_new_name [id=f29a2b35a7af48659aea91b865e44a33]) successfully deleted
I20240503 13:59:42.380509 10751 raft_consensus.cc:482] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:42.380626 10751 raft_consensus.cc:504] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:42.380920 10862 raft_consensus.cc:482] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:42.381057 10862 raft_consensus.cc:504] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:42.381424 10862 leader_election.cc:288] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:42.381460 10751 leader_election.cc:288] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:42.381600 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d38e1d60dd42eabf7191598ded8cd5" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:42.381624 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d38e1d60dd42eabf7191598ded8cd5" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:42.381734 6922 raft_consensus.cc:2454] T 47d38e1d60dd42eabf7191598ded8cd5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:42.381733 9211 raft_consensus.cc:2454] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:42.381873 9342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3dd70554b644cddb1e8eefe7e4ea646" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:42.381994 9342 raft_consensus.cc:2454] T f3dd70554b644cddb1e8eefe7e4ea646 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:42.382011 9278 leader_election.cc:302] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:42.382082 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3dd70554b644cddb1e8eefe7e4ea646" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:42.382218 6658 raft_consensus.cc:2454] T f3dd70554b644cddb1e8eefe7e4ea646 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:42.382275 10751 raft_consensus.cc:2781] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:42.382292 6465 leader_election.cc:302] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:42.382328 10751 raft_consensus.cc:482] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:42.382364 10751 raft_consensus.cc:3037] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:42.382622 10862 raft_consensus.cc:2781] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:42.382681 10862 raft_consensus.cc:482] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:42.382717 10862 raft_consensus.cc:3037] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:42.383523 10751 raft_consensus.cc:504] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:42.383795 10862 raft_consensus.cc:504] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:42.383908 10751 leader_election.cc:288] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:42.384152 10862 leader_election.cc:288] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:42.384188 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d38e1d60dd42eabf7191598ded8cd5" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:42.384261 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d38e1d60dd42eabf7191598ded8cd5" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:42.384348 6922 raft_consensus.cc:3037] T 47d38e1d60dd42eabf7191598ded8cd5 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:42.384379 9211 raft_consensus.cc:3037] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:42.384572 9342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3dd70554b644cddb1e8eefe7e4ea646" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:42.384658 9342 raft_consensus.cc:3037] T f3dd70554b644cddb1e8eefe7e4ea646 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:42.384755 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3dd70554b644cddb1e8eefe7e4ea646" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:42.384860 6658 raft_consensus.cc:3037] T f3dd70554b644cddb1e8eefe7e4ea646 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:42.385360 9211 raft_consensus.cc:2454] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:42.385397 6922 raft_consensus.cc:2454] T 47d38e1d60dd42eabf7191598ded8cd5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:42.385511 9342 raft_consensus.cc:2454] T f3dd70554b644cddb1e8eefe7e4ea646 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:42.385763 9278 leader_election.cc:302] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:42.385784 6465 leader_election.cc:302] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:42.385936 10862 raft_consensus.cc:2781] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:42.385958 10751 raft_consensus.cc:2781] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:42.386004 10862 raft_consensus.cc:686] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Becoming Leader. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Running, Role: LEADER
I20240503 13:59:42.386147 10751 raft_consensus.cc:686] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:42.386166 6658 raft_consensus.cc:2454] T f3dd70554b644cddb1e8eefe7e4ea646 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:42.386144 10862 consensus_queue.cc:237] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:42.386281 10751 consensus_queue.cc:237] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:42.389662 6379 catalog_manager.cc:5484] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:42.389766 6386 catalog_manager.cc:5484] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:42.436033 6386 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:42.442252 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 0860356c9daf42bfa20efb2afdfb70e2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3d8bf6e4fbec4af7940f8137d73e8177]), partition=
I20240503 13:59:42.442409 6900 tablet_service.cc:1467] Processing CreateTablet for tablet 0860356c9daf42bfa20efb2afdfb70e2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3d8bf6e4fbec4af7940f8137d73e8177]), partition=
I20240503 13:59:42.442567 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0860356c9daf42bfa20efb2afdfb70e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:42.442567 9190 tablet_service.cc:1467] Processing CreateTablet for tablet 0860356c9daf42bfa20efb2afdfb70e2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3d8bf6e4fbec4af7940f8137d73e8177]), partition=
I20240503 13:59:42.442654 6900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0860356c9daf42bfa20efb2afdfb70e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:42.442781 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0860356c9daf42bfa20efb2afdfb70e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:42.444415 11002 tablet_bootstrap.cc:492] T 0860356c9daf42bfa20efb2afdfb70e2 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:42.444917 11001 tablet_bootstrap.cc:492] T 0860356c9daf42bfa20efb2afdfb70e2 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:42.445183 11017 tablet_bootstrap.cc:492] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:42.445297 11002 tablet_bootstrap.cc:654] T 0860356c9daf42bfa20efb2afdfb70e2 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:42.445909 11001 tablet_bootstrap.cc:654] T 0860356c9daf42bfa20efb2afdfb70e2 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:42.446153 11017 tablet_bootstrap.cc:654] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:42.447136 11001 tablet_bootstrap.cc:492] T 0860356c9daf42bfa20efb2afdfb70e2 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:42.447221 11001 ts_tablet_manager.cc:1367] T 0860356c9daf42bfa20efb2afdfb70e2 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:42.447427 11017 tablet_bootstrap.cc:492] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:42.447522 11017 ts_tablet_manager.cc:1367] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:42.447676 11001 raft_consensus.cc:348] T 0860356c9daf42bfa20efb2afdfb70e2 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:42.447793 11001 raft_consensus.cc:374] T 0860356c9daf42bfa20efb2afdfb70e2 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:42.447829 11001 raft_consensus.cc:729] T 0860356c9daf42bfa20efb2afdfb70e2 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:42.447961 11001 consensus_queue.cc:260] T 0860356c9daf42bfa20efb2afdfb70e2 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:42.448071 11017 raft_consensus.cc:348] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:42.448189 11017 raft_consensus.cc:374] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:42.448228 11017 raft_consensus.cc:729] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:42.448261 11001 ts_tablet_manager.cc:1398] T 0860356c9daf42bfa20efb2afdfb70e2 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:42.448344 11017 consensus_queue.cc:260] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:42.448705 11017 ts_tablet_manager.cc:1398] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:42.449678 11002 tablet_bootstrap.cc:492] T 0860356c9daf42bfa20efb2afdfb70e2 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:42.449765 11002 ts_tablet_manager.cc:1367] T 0860356c9daf42bfa20efb2afdfb70e2 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:42.450224 11002 raft_consensus.cc:348] T 0860356c9daf42bfa20efb2afdfb70e2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:42.450341 11002 raft_consensus.cc:374] T 0860356c9daf42bfa20efb2afdfb70e2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:42.450376 11002 raft_consensus.cc:729] T 0860356c9daf42bfa20efb2afdfb70e2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:42.450487 11002 consensus_queue.cc:260] T 0860356c9daf42bfa20efb2afdfb70e2 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:42.450744 11002 ts_tablet_manager.cc:1398] T 0860356c9daf42bfa20efb2afdfb70e2 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:42.543017 10769 raft_consensus.cc:482] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:42.543177 10769 raft_consensus.cc:504] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:42.543717 10769 leader_election.cc:288] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:42.543991 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0860356c9daf42bfa20efb2afdfb70e2" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:42.544138 9211 raft_consensus.cc:2454] T 0860356c9daf42bfa20efb2afdfb70e2 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:42.544179 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0860356c9daf42bfa20efb2afdfb70e2" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:42.544317 6922 raft_consensus.cc:2454] T 0860356c9daf42bfa20efb2afdfb70e2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:42.544401 8883 leader_election.cc:302] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:42.544559 10769 raft_consensus.cc:2781] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:42.544620 10769 raft_consensus.cc:482] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:42.544665 10769 raft_consensus.cc:3037] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:42.545682 10769 raft_consensus.cc:504] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:42.545975 10769 leader_election.cc:288] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:42.546308 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0860356c9daf42bfa20efb2afdfb70e2" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:42.546339 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0860356c9daf42bfa20efb2afdfb70e2" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:42.546401 9211 raft_consensus.cc:3037] T 0860356c9daf42bfa20efb2afdfb70e2 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:42.546469 6922 raft_consensus.cc:3037] T 0860356c9daf42bfa20efb2afdfb70e2 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:42.547140 9211 raft_consensus.cc:2454] T 0860356c9daf42bfa20efb2afdfb70e2 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:42.547217 6922 raft_consensus.cc:2454] T 0860356c9daf42bfa20efb2afdfb70e2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:42.547492 8883 leader_election.cc:302] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:42.547744 10769 raft_consensus.cc:2781] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:42.547825 10769 raft_consensus.cc:686] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 13:59:42.547968 10769 consensus_queue.cc:237] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:42.549968 6386 catalog_manager.cc:5484] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:42.669610 6386 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:42.673480 6920 raft_consensus.cc:1261] T 0860356c9daf42bfa20efb2afdfb70e2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:42.673501 9211 raft_consensus.cc:1261] T 0860356c9daf42bfa20efb2afdfb70e2 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:42.673874 10769 consensus_queue.cc:1035] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:42.674062 10981 consensus_queue.cc:1035] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:42.677448 6386 catalog_manager.cc:5576] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531) does not have the latest schema for tablet 0860356c9daf42bfa20efb2afdfb70e2 (table rebalancer_dynamic_table [id=3d8bf6e4fbec4af7940f8137d73e8177]). Expected version 1 got 0
I20240503 13:59:42.678081 6379 catalog_manager.cc:5576] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091) does not have the latest schema for tablet 0860356c9daf42bfa20efb2afdfb70e2 (table rebalancer_dynamic_table [id=3d8bf6e4fbec4af7940f8137d73e8177]). Expected version 1 got 0
I20240503 13:59:42.680480 10951 tablet.cc:1722] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:42.680806 10951 tablet.cc:1620] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:42.681186 11028 tablet.cc:1722] T 0860356c9daf42bfa20efb2afdfb70e2 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:42.681505 11028 tablet.cc:1620] T 0860356c9daf42bfa20efb2afdfb70e2 P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:42.682080 11029 tablet.cc:1722] T 0860356c9daf42bfa20efb2afdfb70e2 P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:42.682350 11029 tablet.cc:1620] T 0860356c9daf42bfa20efb2afdfb70e2 P f58065db70b64ef8b1dc499aa6ae4460: MemRowSet was empty: no flush needed.
I20240503 13:59:42.683748 6379 catalog_manager.cc:5576] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091) does not have the latest schema for tablet 0860356c9daf42bfa20efb2afdfb70e2 (table rebalancer_dynamic_table [id=3d8bf6e4fbec4af7940f8137d73e8177]). Expected version 1 got 0
I20240503 13:59:42.686478 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=3d8bf6e4fbec4af7940f8137d73e8177] alter complete (version 1)
I20240503 13:59:42.691872 10564 consensus_queue.cc:1035] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:42.696512 10751 consensus_queue.cc:1035] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:42.784289 10564 consensus_queue.cc:1035] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:42.788147 10751 consensus_queue.cc:1035] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:42.791189 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:42.796631 10951 tablet.cc:1722] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:42.796947 10951 tablet.cc:1620] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:42.797973 11029 tablet.cc:1722] T 0860356c9daf42bfa20efb2afdfb70e2 P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:42.798403 11029 tablet.cc:1620] T 0860356c9daf42bfa20efb2afdfb70e2 P f58065db70b64ef8b1dc499aa6ae4460: MemRowSet was empty: no flush needed.
I20240503 13:59:42.798759 11028 tablet.cc:1722] T 0860356c9daf42bfa20efb2afdfb70e2 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:42.799036 11028 tablet.cc:1620] T 0860356c9daf42bfa20efb2afdfb70e2 P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:42.803920 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=3d8bf6e4fbec4af7940f8137d73e8177] alter complete (version 2)
I20240503 13:59:42.906162 6386 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:42.911391 10951 tablet.cc:1722] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:42.912760 11028 tablet.cc:1722] T 0860356c9daf42bfa20efb2afdfb70e2 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:42.913780 11029 tablet.cc:1722] T 0860356c9daf42bfa20efb2afdfb70e2 P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:42.918452 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=3d8bf6e4fbec4af7940f8137d73e8177] alter complete (version 3)
I20240503 13:59:42.925283 6380 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:42.932591 9060 tablet_service.cc:1467] Processing CreateTablet for tablet 96db3ad20b864deb83dc3e2d64e2787e (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=7a1d16a1055e477183d59c2aa87d96fb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:42.932837 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 96db3ad20b864deb83dc3e2d64e2787e (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=7a1d16a1055e477183d59c2aa87d96fb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:42.932940 9060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96db3ad20b864deb83dc3e2d64e2787e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:42.933135 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96db3ad20b864deb83dc3e2d64e2787e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:42.933840 6771 tablet_service.cc:1467] Processing CreateTablet for tablet 96db3ad20b864deb83dc3e2d64e2787e (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=7a1d16a1055e477183d59c2aa87d96fb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:42.934124 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96db3ad20b864deb83dc3e2d64e2787e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:42.935068 11037 tablet_bootstrap.cc:492] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:42.935120 6638 tablet_service.cc:1467] Processing CreateTablet for tablet ea438ff21d9744f697878406493f0fdc (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=7a1d16a1055e477183d59c2aa87d96fb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:42.935353 6638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea438ff21d9744f697878406493f0fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:42.935528 6639 tablet_service.cc:1467] Processing CreateTablet for tablet ef19b1e5f15e49448c1b2919a24090d3 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=7a1d16a1055e477183d59c2aa87d96fb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:42.935740 6639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef19b1e5f15e49448c1b2919a24090d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:42.936614 11037 tablet_bootstrap.cc:654] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:42.936784 6769 tablet_service.cc:1467] Processing CreateTablet for tablet ea438ff21d9744f697878406493f0fdc (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=7a1d16a1055e477183d59c2aa87d96fb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:42.937016 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea438ff21d9744f697878406493f0fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:42.938354 8928 tablet_service.cc:1467] Processing CreateTablet for tablet ef19b1e5f15e49448c1b2919a24090d3 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=7a1d16a1055e477183d59c2aa87d96fb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:42.938633 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef19b1e5f15e49448c1b2919a24090d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:42.939612 11038 tablet_bootstrap.cc:492] T 96db3ad20b864deb83dc3e2d64e2787e P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:42.940747 11038 tablet_bootstrap.cc:654] T 96db3ad20b864deb83dc3e2d64e2787e P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:42.941064 11017 tablet_bootstrap.cc:492] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:42.941912 11017 tablet_bootstrap.cc:654] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:42.942283 6508 tablet_service.cc:1467] Processing CreateTablet for tablet ea438ff21d9744f697878406493f0fdc (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=7a1d16a1055e477183d59c2aa87d96fb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:42.942529 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea438ff21d9744f697878406493f0fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:42.942615 6507 tablet_service.cc:1467] Processing CreateTablet for tablet ef19b1e5f15e49448c1b2919a24090d3 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=7a1d16a1055e477183d59c2aa87d96fb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:42.942828 6507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef19b1e5f15e49448c1b2919a24090d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:42.944674 11043 tablet_bootstrap.cc:492] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:42.945052 11017 tablet_bootstrap.cc:492] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:42.945134 11017 ts_tablet_manager.cc:1367] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 13:59:42.945564 11017 raft_consensus.cc:348] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:42.945670 11043 tablet_bootstrap.cc:654] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:42.945672 11017 raft_consensus.cc:374] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:42.945704 11017 raft_consensus.cc:729] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:42.945818 11017 consensus_queue.cc:260] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:42.946065 11017 ts_tablet_manager.cc:1398] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:42.947445 11039 tablet_bootstrap.cc:492] T 96db3ad20b864deb83dc3e2d64e2787e P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:42.948045 11043 tablet_bootstrap.cc:492] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:42.948137 11043 ts_tablet_manager.cc:1367] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:42.948644 11039 tablet_bootstrap.cc:654] T 96db3ad20b864deb83dc3e2d64e2787e P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:42.948638 11043 raft_consensus.cc:348] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:42.948776 11043 raft_consensus.cc:374] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:42.948814 11043 raft_consensus.cc:729] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:42.948927 11043 consensus_queue.cc:260] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:42.949132 11037 tablet_bootstrap.cc:492] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:42.949220 11037 ts_tablet_manager.cc:1367] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 13:59:42.949265 11043 ts_tablet_manager.cc:1398] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:42.949424 11043 tablet_bootstrap.cc:492] T ef19b1e5f15e49448c1b2919a24090d3 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:42.949757 11037 raft_consensus.cc:348] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:42.949860 11037 raft_consensus.cc:374] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:42.950770 11043 tablet_bootstrap.cc:654] T ef19b1e5f15e49448c1b2919a24090d3 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:42.951377 11038 tablet_bootstrap.cc:492] T 96db3ad20b864deb83dc3e2d64e2787e P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:42.951531 11038 ts_tablet_manager.cc:1367] T 96db3ad20b864deb83dc3e2d64e2787e P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 13:59:42.951905 11037 raft_consensus.cc:729] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:42.952217 11038 raft_consensus.cc:348] T 96db3ad20b864deb83dc3e2d64e2787e P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:42.952337 11038 raft_consensus.cc:374] T 96db3ad20b864deb83dc3e2d64e2787e P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:42.952416 11038 raft_consensus.cc:729] T 96db3ad20b864deb83dc3e2d64e2787e P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:42.952566 11038 consensus_queue.cc:260] T 96db3ad20b864deb83dc3e2d64e2787e P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:42.952911 11038 ts_tablet_manager.cc:1398] T 96db3ad20b864deb83dc3e2d64e2787e P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:42.953315 11038 tablet_bootstrap.cc:492] T ea438ff21d9744f697878406493f0fdc P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:42.954411 11038 tablet_bootstrap.cc:654] T ea438ff21d9744f697878406493f0fdc P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:42.954895 11043 tablet_bootstrap.cc:492] T ef19b1e5f15e49448c1b2919a24090d3 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:42.955004 11043 ts_tablet_manager.cc:1367] T ef19b1e5f15e49448c1b2919a24090d3 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:59:42.955643 11043 raft_consensus.cc:348] T ef19b1e5f15e49448c1b2919a24090d3 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:42.955772 11043 raft_consensus.cc:374] T ef19b1e5f15e49448c1b2919a24090d3 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:42.955838 11043 raft_consensus.cc:729] T ef19b1e5f15e49448c1b2919a24090d3 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:42.952607 11037 consensus_queue.cc:260] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:42.956000 11043 consensus_queue.cc:260] T ef19b1e5f15e49448c1b2919a24090d3 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:42.956638 11038 tablet_bootstrap.cc:492] T ea438ff21d9744f697878406493f0fdc P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:42.956728 11038 ts_tablet_manager.cc:1367] T ea438ff21d9744f697878406493f0fdc P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:42.957161 11038 raft_consensus.cc:348] T ea438ff21d9744f697878406493f0fdc P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:42.957295 11038 raft_consensus.cc:374] T ea438ff21d9744f697878406493f0fdc P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:42.957353 11038 raft_consensus.cc:729] T ea438ff21d9744f697878406493f0fdc P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:42.957484 11038 consensus_queue.cc:260] T ea438ff21d9744f697878406493f0fdc P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:42.957804 11038 ts_tablet_manager.cc:1398] T ea438ff21d9744f697878406493f0fdc P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:42.958146 11038 tablet_bootstrap.cc:492] T ef19b1e5f15e49448c1b2919a24090d3 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:42.958443 11043 ts_tablet_manager.cc:1398] T ef19b1e5f15e49448c1b2919a24090d3 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:42.959717 11038 tablet_bootstrap.cc:654] T ef19b1e5f15e49448c1b2919a24090d3 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:42.959898 11037 ts_tablet_manager.cc:1398] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 13:59:42.961048 11039 tablet_bootstrap.cc:492] T 96db3ad20b864deb83dc3e2d64e2787e P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:42.961155 11039 ts_tablet_manager.cc:1367] T 96db3ad20b864deb83dc3e2d64e2787e P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 13:59:42.961216 11038 tablet_bootstrap.cc:492] T ef19b1e5f15e49448c1b2919a24090d3 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:42.961300 11038 ts_tablet_manager.cc:1367] T ef19b1e5f15e49448c1b2919a24090d3 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:42.961645 11039 raft_consensus.cc:348] T 96db3ad20b864deb83dc3e2d64e2787e P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:42.961771 11039 raft_consensus.cc:374] T 96db3ad20b864deb83dc3e2d64e2787e P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:42.961715 11038 raft_consensus.cc:348] T ef19b1e5f15e49448c1b2919a24090d3 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:42.961802 11039 raft_consensus.cc:729] T 96db3ad20b864deb83dc3e2d64e2787e P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:42.961825 11038 raft_consensus.cc:374] T ef19b1e5f15e49448c1b2919a24090d3 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:42.961853 11038 raft_consensus.cc:729] T ef19b1e5f15e49448c1b2919a24090d3 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:42.961926 11039 consensus_queue.cc:260] T 96db3ad20b864deb83dc3e2d64e2787e P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:42.961979 11038 consensus_queue.cc:260] T ef19b1e5f15e49448c1b2919a24090d3 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:42.962374 11038 ts_tablet_manager.cc:1398] T ef19b1e5f15e49448c1b2919a24090d3 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:42.964711 11039 ts_tablet_manager.cc:1398] T 96db3ad20b864deb83dc3e2d64e2787e P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:59:42.964883 11039 tablet_bootstrap.cc:492] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:42.965947 11039 tablet_bootstrap.cc:654] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:42.967159 11039 tablet_bootstrap.cc:492] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:42.967274 11039 ts_tablet_manager.cc:1367] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:42.968124 11039 raft_consensus.cc:348] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:42.968305 11039 raft_consensus.cc:374] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:42.968391 11039 raft_consensus.cc:729] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:42.968525 11039 consensus_queue.cc:260] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:42.968830 11039 ts_tablet_manager.cc:1398] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:42.976120 11010 consensus_queue.cc:1035] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:42.980460 11052 consensus_queue.cc:1035] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:42.983537 10624 raft_consensus.cc:482] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:42.983637 10624 raft_consensus.cc:504] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:42.984117 10624 leader_election.cc:288] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:42.984575 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96db3ad20b864deb83dc3e2d64e2787e" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:42.984704 6658 raft_consensus.cc:2454] T 96db3ad20b864deb83dc3e2d64e2787e P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:42.984759 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96db3ad20b864deb83dc3e2d64e2787e" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:42.984905 6790 raft_consensus.cc:2454] T 96db3ad20b864deb83dc3e2d64e2787e P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:42.984959 9017 leader_election.cc:302] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:42.985122 10624 raft_consensus.cc:2781] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:42.985195 10624 raft_consensus.cc:482] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:42.985242 10624 raft_consensus.cc:3037] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:42.986158 10624 raft_consensus.cc:504] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:42.986466 10624 leader_election.cc:288] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:42.986764 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96db3ad20b864deb83dc3e2d64e2787e" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:42.986893 6790 raft_consensus.cc:3037] T 96db3ad20b864deb83dc3e2d64e2787e P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:42.986945 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96db3ad20b864deb83dc3e2d64e2787e" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:42.987099 6658 raft_consensus.cc:3037] T 96db3ad20b864deb83dc3e2d64e2787e P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:42.987759 6790 raft_consensus.cc:2454] T 96db3ad20b864deb83dc3e2d64e2787e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:42.987989 6658 raft_consensus.cc:2454] T 96db3ad20b864deb83dc3e2d64e2787e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:42.988042 9015 leader_election.cc:302] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:42.988199 10624 raft_consensus.cc:2781] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:42.988286 10624 raft_consensus.cc:686] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 13:59:42.988430 10624 consensus_queue.cc:237] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:42.990828 6380 catalog_manager.cc:5484] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:43.015430 10981 raft_consensus.cc:482] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:43.015558 10981 raft_consensus.cc:504] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:43.016041 10981 leader_election.cc:288] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:43.016383 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef19b1e5f15e49448c1b2919a24090d3" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:43.016543 6658 raft_consensus.cc:2454] T ef19b1e5f15e49448c1b2919a24090d3 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:43.016546 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef19b1e5f15e49448c1b2919a24090d3" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:43.016701 6527 raft_consensus.cc:2454] T ef19b1e5f15e49448c1b2919a24090d3 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:43.016873 8884 leader_election.cc:302] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 2145d15ab2d74c0287839266f1411a43; no voters:
I20240503 13:59:43.017359 10981 raft_consensus.cc:2781] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:43.017428 10981 raft_consensus.cc:482] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:43.017470 10981 raft_consensus.cc:3037] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:43.018597 10981 raft_consensus.cc:504] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:43.018918 10981 leader_election.cc:288] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Requested vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:43.019186 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef19b1e5f15e49448c1b2919a24090d3" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:43.019335 6658 raft_consensus.cc:3037] T ef19b1e5f15e49448c1b2919a24090d3 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:43.019340 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef19b1e5f15e49448c1b2919a24090d3" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:43.019456 6527 raft_consensus.cc:3037] T ef19b1e5f15e49448c1b2919a24090d3 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:43.020318 6658 raft_consensus.cc:2454] T ef19b1e5f15e49448c1b2919a24090d3 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:43.020440 6527 raft_consensus.cc:2454] T ef19b1e5f15e49448c1b2919a24090d3 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:43.020684 8884 leader_election.cc:302] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 2145d15ab2d74c0287839266f1411a43; no voters:
I20240503 13:59:43.020876 10981 raft_consensus.cc:2781] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:43.020982 10981 raft_consensus.cc:686] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 13:59:43.021121 10981 consensus_queue.cc:237] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:43.022774 6383 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:43.022908 6383 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:43.023274 6380 catalog_manager.cc:5484] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:43.025259 6383 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 0860356c9daf42bfa20efb2afdfb70e2
I20240503 13:59:43.025854 9190 tablet_service.cc:1514] Processing DeleteTablet for tablet 0860356c9daf42bfa20efb2afdfb70e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:43 UTC) from {username='slave'} at 127.0.0.1:57960
I20240503 13:59:43.025913 8928 tablet_service.cc:1514] Processing DeleteTablet for tablet 0860356c9daf42bfa20efb2afdfb70e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:43 UTC) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:43.026016 6900 tablet_service.cc:1514] Processing DeleteTablet for tablet 0860356c9daf42bfa20efb2afdfb70e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:43 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:43.026376 11059 tablet_replica.cc:330] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:43.026479 11060 tablet_replica.cc:330] T 0860356c9daf42bfa20efb2afdfb70e2 P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:43.026624 11059 raft_consensus.cc:2229] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:43.026790 11060 raft_consensus.cc:2229] T 0860356c9daf42bfa20efb2afdfb70e2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:43.026888 11059 raft_consensus.cc:2258] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:43.026980 11060 raft_consensus.cc:2258] T 0860356c9daf42bfa20efb2afdfb70e2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:43.027575 11058 tablet_replica.cc:330] T 0860356c9daf42bfa20efb2afdfb70e2 P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 13:59:43.027626 11059 ts_tablet_manager.cc:1875] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:43.027763 11058 raft_consensus.cc:2229] T 0860356c9daf42bfa20efb2afdfb70e2 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:43.027923 11058 raft_consensus.cc:2258] T 0860356c9daf42bfa20efb2afdfb70e2 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:43.029330 11060 ts_tablet_manager.cc:1875] T 0860356c9daf42bfa20efb2afdfb70e2 P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:43.030112 11059 ts_tablet_manager.cc:1888] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:43.030231 11059 log.cc:1196] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/0860356c9daf42bfa20efb2afdfb70e2
I20240503 13:59:43.030607 11059 ts_tablet_manager.cc:1909] T 0860356c9daf42bfa20efb2afdfb70e2 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting consensus metadata
I20240503 13:59:43.030651 11058 ts_tablet_manager.cc:1875] T 0860356c9daf42bfa20efb2afdfb70e2 P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:43.031769 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 0860356c9daf42bfa20efb2afdfb70e2 (table rebalancer_dynamic_table_new_name [id=3d8bf6e4fbec4af7940f8137d73e8177]) successfully deleted
I20240503 13:59:43.033526 11058 ts_tablet_manager.cc:1888] T 0860356c9daf42bfa20efb2afdfb70e2 P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:43.033596 11058 log.cc:1196] T 0860356c9daf42bfa20efb2afdfb70e2 P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/0860356c9daf42bfa20efb2afdfb70e2
I20240503 13:59:43.033856 11060 ts_tablet_manager.cc:1888] T 0860356c9daf42bfa20efb2afdfb70e2 P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:43.033877 11058 ts_tablet_manager.cc:1909] T 0860356c9daf42bfa20efb2afdfb70e2 P ab1d21935d6049888eb0ceccc3c9d735: Deleting consensus metadata
I20240503 13:59:43.033942 11060 log.cc:1196] T 0860356c9daf42bfa20efb2afdfb70e2 P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/0860356c9daf42bfa20efb2afdfb70e2
I20240503 13:59:43.034353 11060 ts_tablet_manager.cc:1909] T 0860356c9daf42bfa20efb2afdfb70e2 P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:43.034646 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet 0860356c9daf42bfa20efb2afdfb70e2 (table rebalancer_dynamic_table_new_name [id=3d8bf6e4fbec4af7940f8137d73e8177]) successfully deleted
I20240503 13:59:43.034966 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet 0860356c9daf42bfa20efb2afdfb70e2 (table rebalancer_dynamic_table_new_name [id=3d8bf6e4fbec4af7940f8137d73e8177]) successfully deleted
I20240503 13:59:43.127161 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:43.134018 9190 tablet_service.cc:1467] Processing CreateTablet for tablet 4d013744cc8e407fb07ce4c7d5bec276 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fb7501a3dddf4fd7a7a8e48d3e802837]), partition=
I20240503 13:59:43.134176 6769 tablet_service.cc:1467] Processing CreateTablet for tablet 4d013744cc8e407fb07ce4c7d5bec276 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fb7501a3dddf4fd7a7a8e48d3e802837]), partition=
I20240503 13:59:43.134160 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 4d013744cc8e407fb07ce4c7d5bec276 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fb7501a3dddf4fd7a7a8e48d3e802837]), partition=
I20240503 13:59:43.134336 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d013744cc8e407fb07ce4c7d5bec276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:43.134414 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d013744cc8e407fb07ce4c7d5bec276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:43.134414 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d013744cc8e407fb07ce4c7d5bec276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:43.136550 11039 tablet_bootstrap.cc:492] T 4d013744cc8e407fb07ce4c7d5bec276 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:43.136889 11062 tablet_bootstrap.cc:492] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:43.137562 11039 tablet_bootstrap.cc:654] T 4d013744cc8e407fb07ce4c7d5bec276 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:43.137823 11062 tablet_bootstrap.cc:654] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:43.138211 11017 tablet_bootstrap.cc:492] T 4d013744cc8e407fb07ce4c7d5bec276 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:43.139161 11017 tablet_bootstrap.cc:654] T 4d013744cc8e407fb07ce4c7d5bec276 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:43.139398 11062 tablet_bootstrap.cc:492] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:43.139474 11062 ts_tablet_manager.cc:1367] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:43.139925 11039 tablet_bootstrap.cc:492] T 4d013744cc8e407fb07ce4c7d5bec276 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:43.140015 11039 ts_tablet_manager.cc:1367] T 4d013744cc8e407fb07ce4c7d5bec276 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:43.140179 11062 raft_consensus.cc:348] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:43.140298 11062 raft_consensus.cc:374] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:43.140339 11062 raft_consensus.cc:729] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:43.140482 11039 raft_consensus.cc:348] T 4d013744cc8e407fb07ce4c7d5bec276 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:43.140513 11062 consensus_queue.cc:260] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:43.140599 11039 raft_consensus.cc:374] T 4d013744cc8e407fb07ce4c7d5bec276 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:43.140642 11039 raft_consensus.cc:729] T 4d013744cc8e407fb07ce4c7d5bec276 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:43.140756 11039 consensus_queue.cc:260] T 4d013744cc8e407fb07ce4c7d5bec276 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:43.141222 11062 ts_tablet_manager.cc:1398] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:43.142571 11039 ts_tablet_manager.cc:1398] T 4d013744cc8e407fb07ce4c7d5bec276 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:43.143556 11017 tablet_bootstrap.cc:492] T 4d013744cc8e407fb07ce4c7d5bec276 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:43.143657 11017 ts_tablet_manager.cc:1367] T 4d013744cc8e407fb07ce4c7d5bec276 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 13:59:43.144104 11017 raft_consensus.cc:348] T 4d013744cc8e407fb07ce4c7d5bec276 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:43.144243 11017 raft_consensus.cc:374] T 4d013744cc8e407fb07ce4c7d5bec276 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:43.144277 11017 raft_consensus.cc:729] T 4d013744cc8e407fb07ce4c7d5bec276 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:43.144395 11017 consensus_queue.cc:260] T 4d013744cc8e407fb07ce4c7d5bec276 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:43.145629 11017 ts_tablet_manager.cc:1398] T 4d013744cc8e407fb07ce4c7d5bec276 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:43.193645 10942 raft_consensus.cc:482] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:43.193787 10942 raft_consensus.cc:504] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:43.194181 10942 leader_election.cc:288] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:43.194490 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d013744cc8e407fb07ce4c7d5bec276" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:43.194490 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d013744cc8e407fb07ce4c7d5bec276" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:43.194622 6790 raft_consensus.cc:2454] T 4d013744cc8e407fb07ce4c7d5bec276 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:43.194666 8948 raft_consensus.cc:2454] T 4d013744cc8e407fb07ce4c7d5bec276 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:43.194917 9146 leader_election.cc:302] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:43.195134 10942 raft_consensus.cc:2781] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:43.195199 10942 raft_consensus.cc:482] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:43.195235 10942 raft_consensus.cc:3037] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:43.196255 10942 raft_consensus.cc:504] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:43.196520 10942 leader_election.cc:288] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Requested vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:43.196879 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d013744cc8e407fb07ce4c7d5bec276" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:43.196935 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d013744cc8e407fb07ce4c7d5bec276" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:43.197018 8948 raft_consensus.cc:3037] T 4d013744cc8e407fb07ce4c7d5bec276 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:43.197052 6790 raft_consensus.cc:3037] T 4d013744cc8e407fb07ce4c7d5bec276 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:43.197944 6790 raft_consensus.cc:2454] T 4d013744cc8e407fb07ce4c7d5bec276 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:43.197944 8948 raft_consensus.cc:2454] T 4d013744cc8e407fb07ce4c7d5bec276 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:43.198271 9147 leader_election.cc:302] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:43.198417 10942 raft_consensus.cc:2781] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:43.198514 10942 raft_consensus.cc:686] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 13:59:43.198657 10942 consensus_queue.cc:237] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:43.200901 6380 catalog_manager.cc:5484] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:43.287489 10737 raft_consensus.cc:482] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:43.287614 10737 raft_consensus.cc:504] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:43.288120 10737 leader_election.cc:288] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:43.288497 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea438ff21d9744f697878406493f0fdc" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:43.288481 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea438ff21d9744f697878406493f0fdc" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:43.288632 6658 raft_consensus.cc:2454] T ea438ff21d9744f697878406493f0fdc P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:43.288635 6527 raft_consensus.cc:2454] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:43.288913 6727 leader_election.cc:302] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:43.289116 10737 raft_consensus.cc:2781] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:43.289178 10737 raft_consensus.cc:482] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:43.289206 10737 raft_consensus.cc:3037] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:43.290141 10737 raft_consensus.cc:504] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:43.290426 10737 leader_election.cc:288] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:43.290786 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea438ff21d9744f697878406493f0fdc" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:43.290907 6658 raft_consensus.cc:3037] T ea438ff21d9744f697878406493f0fdc P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:43.290891 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea438ff21d9744f697878406493f0fdc" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:43.291030 6527 raft_consensus.cc:3037] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:43.291960 6527 raft_consensus.cc:2454] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:43.291960 6658 raft_consensus.cc:2454] T ea438ff21d9744f697878406493f0fdc P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:43.292309 6727 leader_election.cc:302] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:43.292467 10737 raft_consensus.cc:2781] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:43.292568 10737 raft_consensus.cc:686] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:43.292708 10737 consensus_queue.cc:237] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:43.295487 6380 catalog_manager.cc:5484] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:43.317438 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:43.322331 6790 raft_consensus.cc:1261] T 4d013744cc8e407fb07ce4c7d5bec276 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:43.323618 11022 consensus_queue.cc:1035] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:43.324066 8948 raft_consensus.cc:1261] T 4d013744cc8e407fb07ce4c7d5bec276 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:43.324438 11022 consensus_queue.cc:1035] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:43.328187 6379 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet 4d013744cc8e407fb07ce4c7d5bec276 (table rebalancer_dynamic_table [id=fb7501a3dddf4fd7a7a8e48d3e802837]). Expected version 1 got 0
I20240503 13:59:43.329423 6383 catalog_manager.cc:5576] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365) does not have the latest schema for tablet 4d013744cc8e407fb07ce4c7d5bec276 (table rebalancer_dynamic_table [id=fb7501a3dddf4fd7a7a8e48d3e802837]). Expected version 1 got 0
I20240503 13:59:43.329459 11028 tablet.cc:1722] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:43.329780 11028 tablet.cc:1620] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:43.332684 10951 tablet.cc:1722] T 4d013744cc8e407fb07ce4c7d5bec276 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:43.332989 10951 tablet.cc:1620] T 4d013744cc8e407fb07ce4c7d5bec276 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:43.333257 11072 tablet.cc:1722] T 4d013744cc8e407fb07ce4c7d5bec276 P 4a85e492c8154e9881a9cc2a105d7e64: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:43.333629 11072 tablet.cc:1620] T 4d013744cc8e407fb07ce4c7d5bec276 P 4a85e492c8154e9881a9cc2a105d7e64: MemRowSet was empty: no flush needed.
I20240503 13:59:43.335088 6380 catalog_manager.cc:5576] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365) does not have the latest schema for tablet 4d013744cc8e407fb07ce4c7d5bec276 (table rebalancer_dynamic_table [id=fb7501a3dddf4fd7a7a8e48d3e802837]). Expected version 1 got 0
I20240503 13:59:43.336614 6383 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=fb7501a3dddf4fd7a7a8e48d3e802837] alter complete (version 1)
I20240503 13:59:43.376235 10624 consensus_queue.cc:1035] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:43.380152 10624 consensus_queue.cc:1035] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:43.439028 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:43.445091 11028 tablet.cc:1722] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:43.445436 11028 tablet.cc:1620] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:43.447465 11072 tablet.cc:1722] T 4d013744cc8e407fb07ce4c7d5bec276 P 4a85e492c8154e9881a9cc2a105d7e64: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:43.447744 11072 tablet.cc:1620] T 4d013744cc8e407fb07ce4c7d5bec276 P 4a85e492c8154e9881a9cc2a105d7e64: MemRowSet was empty: no flush needed.
I20240503 13:59:43.454200 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=fb7501a3dddf4fd7a7a8e48d3e802837] alter complete (version 2)
I20240503 13:59:43.459576 10951 tablet.cc:1722] T 4d013744cc8e407fb07ce4c7d5bec276 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:43.463418 10951 tablet.cc:1620] T 4d013744cc8e407fb07ce4c7d5bec276 P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:43.491348 10981 consensus_queue.cc:1035] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:43.497695 10769 consensus_queue.cc:1035] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:43.558774 6383 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:43.564404 11028 tablet.cc:1722] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:43.564518 10951 tablet.cc:1722] T 4d013744cc8e407fb07ce4c7d5bec276 P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:43.565176 11072 tablet.cc:1722] T 4d013744cc8e407fb07ce4c7d5bec276 P 4a85e492c8154e9881a9cc2a105d7e64: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:43.569929 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=fb7501a3dddf4fd7a7a8e48d3e802837] alter complete (version 3)
I20240503 13:59:43.671355 6383 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:43.671520 6383 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:43.673535 6383 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 4d013744cc8e407fb07ce4c7d5bec276
I20240503 13:59:43.674170 8928 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d013744cc8e407fb07ce4c7d5bec276 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:43 UTC) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:43.674165 6769 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d013744cc8e407fb07ce4c7d5bec276 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:43 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:43.674157 9190 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d013744cc8e407fb07ce4c7d5bec276 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:43 UTC) from {username='slave'} at 127.0.0.1:57960
I20240503 13:59:43.674491 11082 tablet_replica.cc:330] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 13:59:43.674510 11081 tablet_replica.cc:330] T 4d013744cc8e407fb07ce4c7d5bec276 P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:43.674643 11082 raft_consensus.cc:2229] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:43.674643 11081 raft_consensus.cc:2229] T 4d013744cc8e407fb07ce4c7d5bec276 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:43.674763 11081 raft_consensus.cc:2258] T 4d013744cc8e407fb07ce4c7d5bec276 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:43.674851 11082 raft_consensus.cc:2258] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:43.674999 11080 tablet_replica.cc:330] T 4d013744cc8e407fb07ce4c7d5bec276 P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:43.675150 11080 raft_consensus.cc:2229] T 4d013744cc8e407fb07ce4c7d5bec276 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:43.675312 11080 raft_consensus.cc:2258] T 4d013744cc8e407fb07ce4c7d5bec276 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:43.675488 11082 ts_tablet_manager.cc:1875] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:43.675791 11081 ts_tablet_manager.cc:1875] T 4d013744cc8e407fb07ce4c7d5bec276 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:43.676259 11080 ts_tablet_manager.cc:1875] T 4d013744cc8e407fb07ce4c7d5bec276 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:43.677984 11082 ts_tablet_manager.cc:1888] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:43.678112 11082 log.cc:1196] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/4d013744cc8e407fb07ce4c7d5bec276
I20240503 13:59:43.678436 11082 ts_tablet_manager.cc:1909] T 4d013744cc8e407fb07ce4c7d5bec276 P ab1d21935d6049888eb0ceccc3c9d735: Deleting consensus metadata
I20240503 13:59:43.679430 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet 4d013744cc8e407fb07ce4c7d5bec276 (table rebalancer_dynamic_table_new_name [id=fb7501a3dddf4fd7a7a8e48d3e802837]) successfully deleted
I20240503 13:59:43.680783 11081 ts_tablet_manager.cc:1888] T 4d013744cc8e407fb07ce4c7d5bec276 P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:43.680869 11081 log.cc:1196] T 4d013744cc8e407fb07ce4c7d5bec276 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/4d013744cc8e407fb07ce4c7d5bec276
I20240503 13:59:43.681124 11081 ts_tablet_manager.cc:1909] T 4d013744cc8e407fb07ce4c7d5bec276 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting consensus metadata
I20240503 13:59:43.681603 11080 ts_tablet_manager.cc:1888] T 4d013744cc8e407fb07ce4c7d5bec276 P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:43.681674 11080 log.cc:1196] T 4d013744cc8e407fb07ce4c7d5bec276 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/4d013744cc8e407fb07ce4c7d5bec276
I20240503 13:59:43.681831 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 4d013744cc8e407fb07ce4c7d5bec276 (table rebalancer_dynamic_table_new_name [id=fb7501a3dddf4fd7a7a8e48d3e802837]) successfully deleted
I20240503 13:59:43.682003 11080 ts_tablet_manager.cc:1909] T 4d013744cc8e407fb07ce4c7d5bec276 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:43.682579 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet 4d013744cc8e407fb07ce4c7d5bec276 (table rebalancer_dynamic_table_new_name [id=fb7501a3dddf4fd7a7a8e48d3e802837]) successfully deleted
I20240503 13:59:43.739077 11034 consensus_queue.cc:1035] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:43.742183 10737 consensus_queue.cc:1035] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:43.775362 6379 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:43.781292 6900 tablet_service.cc:1467] Processing CreateTablet for tablet a187aaf172ac473186c9a1c331536771 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=123262ae9286494f80ff1e03a97db2a7]), partition=
I20240503 13:59:43.781378 9321 tablet_service.cc:1467] Processing CreateTablet for tablet a187aaf172ac473186c9a1c331536771 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=123262ae9286494f80ff1e03a97db2a7]), partition=
I20240503 13:59:43.781533 6507 tablet_service.cc:1467] Processing CreateTablet for tablet a187aaf172ac473186c9a1c331536771 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=123262ae9286494f80ff1e03a97db2a7]), partition=
I20240503 13:59:43.781611 6900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a187aaf172ac473186c9a1c331536771. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:43.781625 9321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a187aaf172ac473186c9a1c331536771. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:43.781742 6507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a187aaf172ac473186c9a1c331536771. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:43.783509 11085 tablet_bootstrap.cc:492] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:43.784420 11087 tablet_bootstrap.cc:492] T a187aaf172ac473186c9a1c331536771 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:43.784515 11085 tablet_bootstrap.cc:654] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:43.785189 11087 tablet_bootstrap.cc:654] T a187aaf172ac473186c9a1c331536771 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:43.785823 11085 tablet_bootstrap.cc:492] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:43.785913 11086 tablet_bootstrap.cc:492] T a187aaf172ac473186c9a1c331536771 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:43.786089 11085 ts_tablet_manager.cc:1367] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:43.786521 11087 tablet_bootstrap.cc:492] T a187aaf172ac473186c9a1c331536771 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:43.786621 11087 ts_tablet_manager.cc:1367] T a187aaf172ac473186c9a1c331536771 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:43.786767 11085 raft_consensus.cc:348] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:43.786895 11085 raft_consensus.cc:374] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:43.786952 11085 raft_consensus.cc:729] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:43.787106 11085 consensus_queue.cc:260] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:43.787106 11087 raft_consensus.cc:348] T a187aaf172ac473186c9a1c331536771 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:43.787358 11087 raft_consensus.cc:374] T a187aaf172ac473186c9a1c331536771 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:43.787422 11087 raft_consensus.cc:729] T a187aaf172ac473186c9a1c331536771 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:43.787477 11085 ts_tablet_manager.cc:1398] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:43.787545 11087 consensus_queue.cc:260] T a187aaf172ac473186c9a1c331536771 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:43.787838 11087 ts_tablet_manager.cc:1398] T a187aaf172ac473186c9a1c331536771 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:43.789187 11086 tablet_bootstrap.cc:654] T a187aaf172ac473186c9a1c331536771 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:43.792670 11086 tablet_bootstrap.cc:492] T a187aaf172ac473186c9a1c331536771 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:43.792805 11086 ts_tablet_manager.cc:1367] T a187aaf172ac473186c9a1c331536771 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:59:43.793428 11086 raft_consensus.cc:348] T a187aaf172ac473186c9a1c331536771 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:43.793560 11086 raft_consensus.cc:374] T a187aaf172ac473186c9a1c331536771 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:43.793601 11086 raft_consensus.cc:729] T a187aaf172ac473186c9a1c331536771 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:43.793726 11086 consensus_queue.cc:260] T a187aaf172ac473186c9a1c331536771 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:43.794142 11086 ts_tablet_manager.cc:1398] T a187aaf172ac473186c9a1c331536771 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:43.823765 6379 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:43.830885 9321 tablet_service.cc:1467] Processing CreateTablet for tablet da57af787aef40628e2c043023763f02 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=550b2587d1ad46df98279b7c7b5fa594]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:43.831107 6640 tablet_service.cc:1467] Processing CreateTablet for tablet f16b677f47f54a13a7f374cce34713bd (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=550b2587d1ad46df98279b7c7b5fa594]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:43.831142 6769 tablet_service.cc:1467] Processing CreateTablet for tablet 387d78763d574e57b914e2dbac83a932 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=550b2587d1ad46df98279b7c7b5fa594]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:43.831228 9321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da57af787aef40628e2c043023763f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:43.831413 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f16b677f47f54a13a7f374cce34713bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:43.831426 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387d78763d574e57b914e2dbac83a932. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:43.834250 8928 tablet_service.cc:1467] Processing CreateTablet for tablet f16b677f47f54a13a7f374cce34713bd (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=550b2587d1ad46df98279b7c7b5fa594]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:43.834378 9190 tablet_service.cc:1467] Processing CreateTablet for tablet da57af787aef40628e2c043023763f02 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=550b2587d1ad46df98279b7c7b5fa594]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:43.834410 9192 tablet_service.cc:1467] Processing CreateTablet for tablet 387d78763d574e57b914e2dbac83a932 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=550b2587d1ad46df98279b7c7b5fa594]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:43.834547 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f16b677f47f54a13a7f374cce34713bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:43.834663 9192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387d78763d574e57b914e2dbac83a932. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:43.835569 9061 tablet_service.cc:1467] Processing CreateTablet for tablet 387d78763d574e57b914e2dbac83a932 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=550b2587d1ad46df98279b7c7b5fa594]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:43.835847 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387d78763d574e57b914e2dbac83a932. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:43.835794 9060 tablet_service.cc:1467] Processing CreateTablet for tablet f16b677f47f54a13a7f374cce34713bd (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=550b2587d1ad46df98279b7c7b5fa594]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:43.836027 9060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f16b677f47f54a13a7f374cce34713bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:43.837301 8926 tablet_service.cc:1467] Processing CreateTablet for tablet da57af787aef40628e2c043023763f02 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=550b2587d1ad46df98279b7c7b5fa594]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:43.837512 8926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da57af787aef40628e2c043023763f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:43.838404 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da57af787aef40628e2c043023763f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:43.839591 11095 tablet_bootstrap.cc:492] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:43.841689 11087 tablet_bootstrap.cc:492] T da57af787aef40628e2c043023763f02 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:43.842455 11094 tablet_bootstrap.cc:492] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:43.843152 11087 tablet_bootstrap.cc:654] T da57af787aef40628e2c043023763f02 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:43.843619 11095 tablet_bootstrap.cc:654] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:43.844295 11093 tablet_bootstrap.cc:492] T f16b677f47f54a13a7f374cce34713bd P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:43.844591 11094 tablet_bootstrap.cc:654] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:43.845106 11095 tablet_bootstrap.cc:492] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:43.845189 11095 ts_tablet_manager.cc:1367] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:59:43.845672 11093 tablet_bootstrap.cc:654] T f16b677f47f54a13a7f374cce34713bd P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:43.845614 11095 raft_consensus.cc:348] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:43.845744 11095 raft_consensus.cc:374] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:43.845777 11095 raft_consensus.cc:729] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:43.845890 11095 consensus_queue.cc:260] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:43.846144 11095 ts_tablet_manager.cc:1398] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:43.846293 11095 tablet_bootstrap.cc:492] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:43.846410 11092 tablet_bootstrap.cc:492] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:43.846539 11094 tablet_bootstrap.cc:492] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:43.846607 11094 ts_tablet_manager.cc:1367] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:43.847092 11094 raft_consensus.cc:348] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:43.847198 11094 raft_consensus.cc:374] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:43.847229 11094 raft_consensus.cc:729] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:43.847363 11091 tablet_bootstrap.cc:492] T f16b677f47f54a13a7f374cce34713bd P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:43.847360 11094 consensus_queue.cc:260] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:43.847656 11092 tablet_bootstrap.cc:654] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:43.847673 11094 ts_tablet_manager.cc:1398] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:43.847848 11094 tablet_bootstrap.cc:492] T da57af787aef40628e2c043023763f02 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:43.848392 11091 tablet_bootstrap.cc:654] T f16b677f47f54a13a7f374cce34713bd P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:43.848754 11094 tablet_bootstrap.cc:654] T da57af787aef40628e2c043023763f02 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:43.849989 11094 tablet_bootstrap.cc:492] T da57af787aef40628e2c043023763f02 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:43.850121 11094 ts_tablet_manager.cc:1367] T da57af787aef40628e2c043023763f02 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:43.850463 11087 tablet_bootstrap.cc:492] T da57af787aef40628e2c043023763f02 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:43.850551 11087 ts_tablet_manager.cc:1367] T da57af787aef40628e2c043023763f02 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:59:43.850595 11094 raft_consensus.cc:348] T da57af787aef40628e2c043023763f02 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:43.850692 11094 raft_consensus.cc:374] T da57af787aef40628e2c043023763f02 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:43.850720 11094 raft_consensus.cc:729] T da57af787aef40628e2c043023763f02 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:43.850840 11094 consensus_queue.cc:260] T da57af787aef40628e2c043023763f02 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:43.851130 11094 ts_tablet_manager.cc:1398] T da57af787aef40628e2c043023763f02 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:43.851572 11087 raft_consensus.cc:348] T da57af787aef40628e2c043023763f02 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:43.851967 11087 raft_consensus.cc:374] T da57af787aef40628e2c043023763f02 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:43.852033 11087 raft_consensus.cc:729] T da57af787aef40628e2c043023763f02 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:43.852172 11087 consensus_queue.cc:260] T da57af787aef40628e2c043023763f02 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:43.852509 11087 ts_tablet_manager.cc:1398] T da57af787aef40628e2c043023763f02 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:43.853286 11093 tablet_bootstrap.cc:492] T f16b677f47f54a13a7f374cce34713bd P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:43.853423 11093 ts_tablet_manager.cc:1367] T f16b677f47f54a13a7f374cce34713bd P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:59:43.853478 11092 tablet_bootstrap.cc:492] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:43.853613 11092 ts_tablet_manager.cc:1367] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:59:43.853745 11091 tablet_bootstrap.cc:492] T f16b677f47f54a13a7f374cce34713bd P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:43.853837 11091 ts_tablet_manager.cc:1367] T f16b677f47f54a13a7f374cce34713bd P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:59:43.854017 11093 raft_consensus.cc:348] T f16b677f47f54a13a7f374cce34713bd P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:43.854125 11093 raft_consensus.cc:374] T f16b677f47f54a13a7f374cce34713bd P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:43.854066 11092 raft_consensus.cc:348] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:43.854154 11093 raft_consensus.cc:729] T f16b677f47f54a13a7f374cce34713bd P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:43.854161 11092 raft_consensus.cc:374] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:43.854192 11092 raft_consensus.cc:729] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:43.854283 11093 consensus_queue.cc:260] T f16b677f47f54a13a7f374cce34713bd P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:43.854341 11092 consensus_queue.cc:260] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:43.854584 11093 ts_tablet_manager.cc:1398] T f16b677f47f54a13a7f374cce34713bd P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:43.854598 11092 ts_tablet_manager.cc:1398] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:43.854863 11093 tablet_bootstrap.cc:492] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:43.856102 11093 tablet_bootstrap.cc:654] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:43.856395 11091 raft_consensus.cc:348] T f16b677f47f54a13a7f374cce34713bd P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:43.856585 11091 raft_consensus.cc:374] T f16b677f47f54a13a7f374cce34713bd P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:43.856671 11091 raft_consensus.cc:729] T f16b677f47f54a13a7f374cce34713bd P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:43.856834 11091 consensus_queue.cc:260] T f16b677f47f54a13a7f374cce34713bd P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:43.857290 11091 ts_tablet_manager.cc:1398] T f16b677f47f54a13a7f374cce34713bd P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:43.860466 11095 tablet_bootstrap.cc:654] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:43.861989 11095 tablet_bootstrap.cc:492] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:43.862098 11095 ts_tablet_manager.cc:1367] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.001s
I20240503 13:59:43.862596 11095 raft_consensus.cc:348] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:43.862700 11095 raft_consensus.cc:374] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:43.862743 11095 raft_consensus.cc:729] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:43.862866 11095 consensus_queue.cc:260] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:43.863114 11095 ts_tablet_manager.cc:1398] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:43.863718 11093 tablet_bootstrap.cc:492] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:43.863955 11093 ts_tablet_manager.cc:1367] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.001s
I20240503 13:59:43.864481 11093 raft_consensus.cc:348] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:43.864715 11093 raft_consensus.cc:374] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:43.864805 11093 raft_consensus.cc:729] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:43.865012 11093 consensus_queue.cc:260] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:43.865612 11093 ts_tablet_manager.cc:1398] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:43.914541 11105 raft_consensus.cc:482] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:43.914691 11105 raft_consensus.cc:504] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:43.915308 11105 leader_election.cc:288] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:43.915575 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387d78763d574e57b914e2dbac83a932" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:43.915606 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387d78763d574e57b914e2dbac83a932" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:43.915714 6791 raft_consensus.cc:2454] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:43.915753 9211 raft_consensus.cc:2454] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:43.915972 9015 leader_election.cc:302] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:43.916182 11105 raft_consensus.cc:2781] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:43.916227 11105 raft_consensus.cc:482] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:43.916257 11105 raft_consensus.cc:3037] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:43.919821 11105 raft_consensus.cc:504] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:43.920243 11105 leader_election.cc:288] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:43.920825 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387d78763d574e57b914e2dbac83a932" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:43.920830 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387d78763d574e57b914e2dbac83a932" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:43.920949 9211 raft_consensus.cc:3037] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:43.920956 6791 raft_consensus.cc:3037] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:43.921819 6791 raft_consensus.cc:2454] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:43.921890 9211 raft_consensus.cc:2454] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:43.922187 9016 leader_election.cc:302] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:43.922392 11105 raft_consensus.cc:2781] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:43.922468 11105 raft_consensus.cc:686] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 13:59:43.922798 11105 consensus_queue.cc:237] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:43.925006 6383 catalog_manager.cc:5484] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:43.958415 11079 raft_consensus.cc:482] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:43.958544 11079 raft_consensus.cc:504] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:43.959089 11079 leader_election.cc:288] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:43.959621 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187aaf172ac473186c9a1c331536771" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:43.959633 9342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187aaf172ac473186c9a1c331536771" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:43.959769 6528 raft_consensus.cc:2454] T a187aaf172ac473186c9a1c331536771 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:43.959780 9342 raft_consensus.cc:2454] T a187aaf172ac473186c9a1c331536771 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:43.960047 6858 leader_election.cc:302] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:43.960251 11079 raft_consensus.cc:2781] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:43.960321 11079 raft_consensus.cc:482] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:43.960376 11079 raft_consensus.cc:3037] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:43.961473 11079 raft_consensus.cc:504] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:43.961962 11079 leader_election.cc:288] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:43.962054 9342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187aaf172ac473186c9a1c331536771" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:43.962160 9342 raft_consensus.cc:3037] T a187aaf172ac473186c9a1c331536771 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:43.962215 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187aaf172ac473186c9a1c331536771" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:43.962337 6528 raft_consensus.cc:3037] T a187aaf172ac473186c9a1c331536771 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:43.963258 9342 raft_consensus.cc:2454] T a187aaf172ac473186c9a1c331536771 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:43.963531 6858 leader_election.cc:302] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:43.963709 11079 raft_consensus.cc:2781] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:43.963994 6528 raft_consensus.cc:2454] T a187aaf172ac473186c9a1c331536771 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:43.964251 11079 raft_consensus.cc:686] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:43.964421 11079 consensus_queue.cc:237] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:43.966841 6380 catalog_manager.cc:5484] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:43.971503 11105 raft_consensus.cc:482] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:43.971599 11105 raft_consensus.cc:504] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:43.972009 11105 leader_election.cc:288] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:43.972353 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f16b677f47f54a13a7f374cce34713bd" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:43.972420 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f16b677f47f54a13a7f374cce34713bd" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:43.972501 8947 raft_consensus.cc:2454] T f16b677f47f54a13a7f374cce34713bd P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:43.972543 6658 raft_consensus.cc:2454] T f16b677f47f54a13a7f374cce34713bd P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:43.972761 9016 leader_election.cc:302] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:43.972980 11105 raft_consensus.cc:2781] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:43.973037 11105 raft_consensus.cc:482] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:43.973074 11105 raft_consensus.cc:3037] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:43.973986 11105 raft_consensus.cc:504] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:43.974279 11105 leader_election.cc:288] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:43.974578 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f16b677f47f54a13a7f374cce34713bd" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:43.974622 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f16b677f47f54a13a7f374cce34713bd" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:43.974709 8947 raft_consensus.cc:3037] T f16b677f47f54a13a7f374cce34713bd P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:43.974722 6658 raft_consensus.cc:3037] T f16b677f47f54a13a7f374cce34713bd P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:43.975526 8947 raft_consensus.cc:2454] T f16b677f47f54a13a7f374cce34713bd P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:43.975566 6658 raft_consensus.cc:2454] T f16b677f47f54a13a7f374cce34713bd P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:43.975816 9016 leader_election.cc:302] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:43.976006 11105 raft_consensus.cc:2781] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:43.976238 11105 raft_consensus.cc:686] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 13:59:43.976379 11105 consensus_queue.cc:237] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:43.978993 6380 catalog_manager.cc:5484] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:44.087008 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:44.091161 9342 raft_consensus.cc:1261] T a187aaf172ac473186c9a1c331536771 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:44.090965 6528 raft_consensus.cc:1261] T a187aaf172ac473186c9a1c331536771 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:44.092134 10904 consensus_queue.cc:1035] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:44.092597 10904 consensus_queue.cc:1035] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:44.095140 6380 catalog_manager.cc:5576] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931) does not have the latest schema for tablet a187aaf172ac473186c9a1c331536771 (table rebalancer_dynamic_table [id=123262ae9286494f80ff1e03a97db2a7]). Expected version 1 got 0
I20240503 13:59:44.095691 11114 tablet.cc:1722] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:44.096051 11114 tablet.cc:1620] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460: MemRowSet was empty: no flush needed.
I20240503 13:59:44.097030 6383 catalog_manager.cc:5576] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021) does not have the latest schema for tablet a187aaf172ac473186c9a1c331536771 (table rebalancer_dynamic_table [id=123262ae9286494f80ff1e03a97db2a7]). Expected version 1 got 0
I20240503 13:59:44.097172 11115 tablet.cc:1722] T a187aaf172ac473186c9a1c331536771 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:44.097451 11115 tablet.cc:1620] T a187aaf172ac473186c9a1c331536771 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:44.099970 6386 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=123262ae9286494f80ff1e03a97db2a7] alter complete (version 1)
I20240503 13:59:44.100032 11117 tablet.cc:1722] T a187aaf172ac473186c9a1c331536771 P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:44.100313 11117 tablet.cc:1620] T a187aaf172ac473186c9a1c331536771 P 5f46e19644fd4f4b825c86ba6af768da: MemRowSet was empty: no flush needed.
I20240503 13:59:44.120172 11061 raft_consensus.cc:482] T da57af787aef40628e2c043023763f02 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:44.120285 11061 raft_consensus.cc:504] T da57af787aef40628e2c043023763f02 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:44.120666 11061 leader_election.cc:288] T da57af787aef40628e2c043023763f02 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:44.120942 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da57af787aef40628e2c043023763f02" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:44.121011 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da57af787aef40628e2c043023763f02" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:44.121075 9211 raft_consensus.cc:2454] T da57af787aef40628e2c043023763f02 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:44.121141 8947 raft_consensus.cc:2454] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:44.121417 9278 leader_election.cc:302] T da57af787aef40628e2c043023763f02 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:44.121665 11061 raft_consensus.cc:2781] T da57af787aef40628e2c043023763f02 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:44.121716 11061 raft_consensus.cc:482] T da57af787aef40628e2c043023763f02 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:44.121755 11061 raft_consensus.cc:3037] T da57af787aef40628e2c043023763f02 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:44.122954 11061 raft_consensus.cc:504] T da57af787aef40628e2c043023763f02 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:44.123304 11061 leader_election.cc:288] T da57af787aef40628e2c043023763f02 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:44.123669 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da57af787aef40628e2c043023763f02" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:44.123728 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da57af787aef40628e2c043023763f02" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:44.123780 9211 raft_consensus.cc:3037] T da57af787aef40628e2c043023763f02 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:44.123848 8947 raft_consensus.cc:3037] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:44.124606 8947 raft_consensus.cc:2454] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:44.124684 9211 raft_consensus.cc:2454] T da57af787aef40628e2c043023763f02 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:44.125020 9278 leader_election.cc:302] T da57af787aef40628e2c043023763f02 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:44.125181 11061 raft_consensus.cc:2781] T da57af787aef40628e2c043023763f02 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:44.125267 11061 raft_consensus.cc:686] T da57af787aef40628e2c043023763f02 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:44.125430 11061 consensus_queue.cc:237] T da57af787aef40628e2c043023763f02 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:44.127995 6386 catalog_manager.cc:5484] T da57af787aef40628e2c043023763f02 P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:44.209010 6386 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:44.214747 11114 tablet.cc:1722] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:44.215091 11114 tablet.cc:1620] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460: MemRowSet was empty: no flush needed.
I20240503 13:59:44.215106 11117 tablet.cc:1722] T a187aaf172ac473186c9a1c331536771 P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:44.215458 11117 tablet.cc:1620] T a187aaf172ac473186c9a1c331536771 P 5f46e19644fd4f4b825c86ba6af768da: MemRowSet was empty: no flush needed.
I20240503 13:59:44.216105 11115 tablet.cc:1722] T a187aaf172ac473186c9a1c331536771 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:44.216392 11115 tablet.cc:1620] T a187aaf172ac473186c9a1c331536771 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:44.223639 6386 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=123262ae9286494f80ff1e03a97db2a7] alter complete (version 2)
I20240503 13:59:44.326061 6385 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:44.331925 11114 tablet.cc:1722] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:44.332787 11117 tablet.cc:1722] T a187aaf172ac473186c9a1c331536771 P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:44.335183 11115 tablet.cc:1722] T a187aaf172ac473186c9a1c331536771 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:44.338938 6385 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=123262ae9286494f80ff1e03a97db2a7] alter complete (version 3)
I20240503 13:59:44.409474 11105 consensus_queue.cc:1035] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:44.414582 10932 consensus_queue.cc:1035] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:44.441884 6385 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:44.442065 6385 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:44.444234 6385 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet a187aaf172ac473186c9a1c331536771
I20240503 13:59:44.444744 9321 tablet_service.cc:1514] Processing DeleteTablet for tablet a187aaf172ac473186c9a1c331536771 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:44 UTC) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:44.444824 6507 tablet_service.cc:1514] Processing DeleteTablet for tablet a187aaf172ac473186c9a1c331536771 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:44 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:44.445061 6900 tablet_service.cc:1514] Processing DeleteTablet for tablet a187aaf172ac473186c9a1c331536771 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:44 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:44.445104 11126 tablet_replica.cc:330] T a187aaf172ac473186c9a1c331536771 P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:44.445256 11126 raft_consensus.cc:2229] T a187aaf172ac473186c9a1c331536771 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:44.445380 11126 raft_consensus.cc:2258] T a187aaf172ac473186c9a1c331536771 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:44.445401 11127 tablet_replica.cc:330] T a187aaf172ac473186c9a1c331536771 P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:44.445544 11127 raft_consensus.cc:2229] T a187aaf172ac473186c9a1c331536771 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:44.445607 11128 tablet_replica.cc:330] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:44.445673 11127 raft_consensus.cc:2258] T a187aaf172ac473186c9a1c331536771 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:44.445960 11128 raft_consensus.cc:2229] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:44.446305 11127 ts_tablet_manager.cc:1875] T a187aaf172ac473186c9a1c331536771 P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:44.446333 11126 ts_tablet_manager.cc:1875] T a187aaf172ac473186c9a1c331536771 P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:44.446350 11128 raft_consensus.cc:2258] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:44.446930 11128 ts_tablet_manager.cc:1875] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:44.448719 11126 ts_tablet_manager.cc:1888] T a187aaf172ac473186c9a1c331536771 P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:44.448825 11126 log.cc:1196] T a187aaf172ac473186c9a1c331536771 P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/a187aaf172ac473186c9a1c331536771
I20240503 13:59:44.448822 11127 ts_tablet_manager.cc:1888] T a187aaf172ac473186c9a1c331536771 P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:44.448984 11127 log.cc:1196] T a187aaf172ac473186c9a1c331536771 P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/a187aaf172ac473186c9a1c331536771
I20240503 13:59:44.449123 11126 ts_tablet_manager.cc:1909] T a187aaf172ac473186c9a1c331536771 P 0d77752181e944698e809e325e5937e6: Deleting consensus metadata
I20240503 13:59:44.449326 11127 ts_tablet_manager.cc:1909] T a187aaf172ac473186c9a1c331536771 P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:44.449772 11128 ts_tablet_manager.cc:1888] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:44.449860 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet a187aaf172ac473186c9a1c331536771 (table rebalancer_dynamic_table_new_name [id=123262ae9286494f80ff1e03a97db2a7]) successfully deleted
I20240503 13:59:44.449986 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet a187aaf172ac473186c9a1c331536771 (table rebalancer_dynamic_table_new_name [id=123262ae9286494f80ff1e03a97db2a7]) successfully deleted
I20240503 13:59:44.450008 11128 log.cc:1196] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/a187aaf172ac473186c9a1c331536771
I20240503 13:59:44.450304 11128 ts_tablet_manager.cc:1909] T a187aaf172ac473186c9a1c331536771 P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:44.450878 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet a187aaf172ac473186c9a1c331536771 (table rebalancer_dynamic_table_new_name [id=123262ae9286494f80ff1e03a97db2a7]) successfully deleted
I20240503 13:59:44.464046 10932 consensus_queue.cc:1035] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:44.467625 11105 consensus_queue.cc:1035] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:44.528537 11061 consensus_queue.cc:1035] T da57af787aef40628e2c043023763f02 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:44.531673 11061 consensus_queue.cc:1035] T da57af787aef40628e2c043023763f02 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:44.545707 6385 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:44.551664 6900 tablet_service.cc:1467] Processing CreateTablet for tablet ed0e74cfbe1a4fbca46c100f3d0d5f8b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1283c7de26ed4f74b345e29414b410a8]), partition=
I20240503 13:59:44.551776 8926 tablet_service.cc:1467] Processing CreateTablet for tablet ed0e74cfbe1a4fbca46c100f3d0d5f8b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1283c7de26ed4f74b345e29414b410a8]), partition=
I20240503 13:59:44.551805 9321 tablet_service.cc:1467] Processing CreateTablet for tablet ed0e74cfbe1a4fbca46c100f3d0d5f8b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1283c7de26ed4f74b345e29414b410a8]), partition=
I20240503 13:59:44.551983 8926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed0e74cfbe1a4fbca46c100f3d0d5f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:44.551985 6900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed0e74cfbe1a4fbca46c100f3d0d5f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:44.552062 9321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed0e74cfbe1a4fbca46c100f3d0d5f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:44.554013 11135 tablet_bootstrap.cc:492] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:44.554478 11137 tablet_bootstrap.cc:492] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:44.555666 11137 tablet_bootstrap.cc:654] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:44.555840 11135 tablet_bootstrap.cc:654] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:44.556023 11136 tablet_bootstrap.cc:492] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:44.557155 11136 tablet_bootstrap.cc:654] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:44.557876 11135 tablet_bootstrap.cc:492] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:44.557972 11135 ts_tablet_manager.cc:1367] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:44.558388 11137 tablet_bootstrap.cc:492] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:44.558476 11137 ts_tablet_manager.cc:1367] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:44.558485 11136 tablet_bootstrap.cc:492] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:44.558483 11135 raft_consensus.cc:348] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:44.558545 11136 ts_tablet_manager.cc:1367] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:44.558596 11135 raft_consensus.cc:374] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:44.558629 11135 raft_consensus.cc:729] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:44.558748 11135 consensus_queue.cc:260] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:44.558948 11137 raft_consensus.cc:348] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:44.559042 11137 raft_consensus.cc:374] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:44.559000 11136 raft_consensus.cc:348] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:44.559074 11137 raft_consensus.cc:729] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:44.559087 11136 raft_consensus.cc:374] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:44.559072 11135 ts_tablet_manager.cc:1398] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:44.559108 11136 raft_consensus.cc:729] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:44.559211 11136 consensus_queue.cc:260] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:44.559213 11137 consensus_queue.cc:260] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:44.559671 11137 ts_tablet_manager.cc:1398] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:44.562335 11136 ts_tablet_manager.cc:1398] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 13:59:44.642982 6380 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:44.650907 6640 tablet_service.cc:1467] Processing CreateTablet for tablet 4748749ca2e94c83a9cff7e2564ba32a (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=06e71db014ed4195a64a38b339ee4ccb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:44.651120 6769 tablet_service.cc:1467] Processing CreateTablet for tablet 4748749ca2e94c83a9cff7e2564ba32a (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=06e71db014ed4195a64a38b339ee4ccb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:44.651261 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4748749ca2e94c83a9cff7e2564ba32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:44.651407 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4748749ca2e94c83a9cff7e2564ba32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:44.652297 6900 tablet_service.cc:1467] Processing CreateTablet for tablet e4fce5deaae4457a9f161e66fcc02c60 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=06e71db014ed4195a64a38b339ee4ccb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:44.652592 6900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4fce5deaae4457a9f161e66fcc02c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:44.653825 6639 tablet_service.cc:1467] Processing CreateTablet for tablet dfa48440020d4754be0f5ddf159c23fe (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=06e71db014ed4195a64a38b339ee4ccb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:44.653825 6901 tablet_service.cc:1467] Processing CreateTablet for tablet dfa48440020d4754be0f5ddf159c23fe (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=06e71db014ed4195a64a38b339ee4ccb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:44.654043 6639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa48440020d4754be0f5ddf159c23fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:44.654055 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa48440020d4754be0f5ddf159c23fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:44.654639 11137 tablet_bootstrap.cc:492] T e4fce5deaae4457a9f161e66fcc02c60 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:44.655635 11137 tablet_bootstrap.cc:654] T e4fce5deaae4457a9f161e66fcc02c60 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:44.656214 6507 tablet_service.cc:1467] Processing CreateTablet for tablet 4748749ca2e94c83a9cff7e2564ba32a (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=06e71db014ed4195a64a38b339ee4ccb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:44.656399 9190 tablet_service.cc:1467] Processing CreateTablet for tablet e4fce5deaae4457a9f161e66fcc02c60 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=06e71db014ed4195a64a38b339ee4ccb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:44.656486 6507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4748749ca2e94c83a9cff7e2564ba32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:44.656646 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4fce5deaae4457a9f161e66fcc02c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:44.657541 9061 tablet_service.cc:1467] Processing CreateTablet for tablet e4fce5deaae4457a9f161e66fcc02c60 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=06e71db014ed4195a64a38b339ee4ccb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:44.657810 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4fce5deaae4457a9f161e66fcc02c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:44.658816 11143 tablet_bootstrap.cc:492] T 4748749ca2e94c83a9cff7e2564ba32a P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:44.659008 9060 tablet_service.cc:1467] Processing CreateTablet for tablet dfa48440020d4754be0f5ddf159c23fe (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=06e71db014ed4195a64a38b339ee4ccb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:44.659237 9060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa48440020d4754be0f5ddf159c23fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:44.659824 11143 tablet_bootstrap.cc:654] T 4748749ca2e94c83a9cff7e2564ba32a P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:44.661149 11143 tablet_bootstrap.cc:492] T 4748749ca2e94c83a9cff7e2564ba32a P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:44.661231 11143 ts_tablet_manager.cc:1367] T 4748749ca2e94c83a9cff7e2564ba32a P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:44.661690 11143 raft_consensus.cc:348] T 4748749ca2e94c83a9cff7e2564ba32a P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:44.661806 11143 raft_consensus.cc:374] T 4748749ca2e94c83a9cff7e2564ba32a P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:44.661840 11143 raft_consensus.cc:729] T 4748749ca2e94c83a9cff7e2564ba32a P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:44.661952 11143 consensus_queue.cc:260] T 4748749ca2e94c83a9cff7e2564ba32a P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:44.662263 11143 ts_tablet_manager.cc:1398] T 4748749ca2e94c83a9cff7e2564ba32a P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:44.662739 11141 tablet_bootstrap.cc:492] T 4748749ca2e94c83a9cff7e2564ba32a P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:44.663154 11142 tablet_bootstrap.cc:492] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:44.663561 11145 tablet_bootstrap.cc:492] T e4fce5deaae4457a9f161e66fcc02c60 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:44.664345 11144 tablet_bootstrap.cc:492] T e4fce5deaae4457a9f161e66fcc02c60 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:44.664885 11137 tablet_bootstrap.cc:492] T e4fce5deaae4457a9f161e66fcc02c60 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:44.664980 11137 ts_tablet_manager.cc:1367] T e4fce5deaae4457a9f161e66fcc02c60 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 13:59:44.665501 11144 tablet_bootstrap.cc:654] T e4fce5deaae4457a9f161e66fcc02c60 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:44.665431 11137 raft_consensus.cc:348] T e4fce5deaae4457a9f161e66fcc02c60 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:44.665544 11137 raft_consensus.cc:374] T e4fce5deaae4457a9f161e66fcc02c60 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:44.665596 11137 raft_consensus.cc:729] T e4fce5deaae4457a9f161e66fcc02c60 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:44.665716 11137 consensus_queue.cc:260] T e4fce5deaae4457a9f161e66fcc02c60 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:44.665827 11141 tablet_bootstrap.cc:654] T 4748749ca2e94c83a9cff7e2564ba32a P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:44.666026 11137 ts_tablet_manager.cc:1398] T e4fce5deaae4457a9f161e66fcc02c60 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:44.666301 11137 tablet_bootstrap.cc:492] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:44.666563 11145 tablet_bootstrap.cc:654] T e4fce5deaae4457a9f161e66fcc02c60 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:44.667340 11137 tablet_bootstrap.cc:654] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:44.667829 11142 tablet_bootstrap.cc:654] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:44.669507 11142 tablet_bootstrap.cc:492] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:44.669554 11137 tablet_bootstrap.cc:492] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:44.669631 11137 ts_tablet_manager.cc:1367] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:59:44.669778 11142 ts_tablet_manager.cc:1367] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:59:44.670413 11142 raft_consensus.cc:348] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:44.670591 11142 raft_consensus.cc:374] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:44.670663 11142 raft_consensus.cc:729] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:44.670835 11142 consensus_queue.cc:260] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:44.671147 11142 ts_tablet_manager.cc:1398] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:44.671365 11142 tablet_bootstrap.cc:492] T dfa48440020d4754be0f5ddf159c23fe P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:44.671396 11137 raft_consensus.cc:348] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:44.671536 11137 raft_consensus.cc:374] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:44.671790 11137 raft_consensus.cc:729] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:44.671972 11137 consensus_queue.cc:260] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:44.672417 11137 ts_tablet_manager.cc:1398] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:59:44.673172 11142 tablet_bootstrap.cc:654] T dfa48440020d4754be0f5ddf159c23fe P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:44.673338 11141 tablet_bootstrap.cc:492] T 4748749ca2e94c83a9cff7e2564ba32a P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:44.673435 11141 ts_tablet_manager.cc:1367] T 4748749ca2e94c83a9cff7e2564ba32a P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 13:59:44.673970 11141 raft_consensus.cc:348] T 4748749ca2e94c83a9cff7e2564ba32a P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:44.674106 11141 raft_consensus.cc:374] T 4748749ca2e94c83a9cff7e2564ba32a P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:44.674167 11141 raft_consensus.cc:729] T 4748749ca2e94c83a9cff7e2564ba32a P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:44.674335 11141 consensus_queue.cc:260] T 4748749ca2e94c83a9cff7e2564ba32a P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:44.674649 11141 ts_tablet_manager.cc:1398] T 4748749ca2e94c83a9cff7e2564ba32a P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:44.675443 11144 tablet_bootstrap.cc:492] T e4fce5deaae4457a9f161e66fcc02c60 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:44.675540 11144 ts_tablet_manager.cc:1367] T e4fce5deaae4457a9f161e66fcc02c60 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 13:59:44.675963 11144 raft_consensus.cc:348] T e4fce5deaae4457a9f161e66fcc02c60 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:44.676092 11144 raft_consensus.cc:374] T e4fce5deaae4457a9f161e66fcc02c60 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:44.676131 11144 raft_consensus.cc:729] T e4fce5deaae4457a9f161e66fcc02c60 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:44.676209 11145 tablet_bootstrap.cc:492] T e4fce5deaae4457a9f161e66fcc02c60 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:44.676296 11145 ts_tablet_manager.cc:1367] T e4fce5deaae4457a9f161e66fcc02c60 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 13:59:44.676244 11144 consensus_queue.cc:260] T e4fce5deaae4457a9f161e66fcc02c60 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:44.676313 11142 tablet_bootstrap.cc:492] T dfa48440020d4754be0f5ddf159c23fe P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:44.676388 11142 ts_tablet_manager.cc:1367] T dfa48440020d4754be0f5ddf159c23fe P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:44.676530 11144 ts_tablet_manager.cc:1398] T e4fce5deaae4457a9f161e66fcc02c60 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:44.676779 11142 raft_consensus.cc:348] T dfa48440020d4754be0f5ddf159c23fe P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:44.676898 11142 raft_consensus.cc:374] T dfa48440020d4754be0f5ddf159c23fe P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:44.676950 11142 raft_consensus.cc:729] T dfa48440020d4754be0f5ddf159c23fe P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:44.676981 11145 raft_consensus.cc:348] T e4fce5deaae4457a9f161e66fcc02c60 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:44.677106 11145 raft_consensus.cc:374] T e4fce5deaae4457a9f161e66fcc02c60 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:44.677170 11145 raft_consensus.cc:729] T e4fce5deaae4457a9f161e66fcc02c60 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:44.677209 11142 consensus_queue.cc:260] T dfa48440020d4754be0f5ddf159c23fe P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:44.677342 11145 consensus_queue.cc:260] T e4fce5deaae4457a9f161e66fcc02c60 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:44.677656 11142 ts_tablet_manager.cc:1398] T dfa48440020d4754be0f5ddf159c23fe P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:44.677786 11145 ts_tablet_manager.cc:1398] T e4fce5deaae4457a9f161e66fcc02c60 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:44.677978 11145 tablet_bootstrap.cc:492] T dfa48440020d4754be0f5ddf159c23fe P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:44.679006 11145 tablet_bootstrap.cc:654] T dfa48440020d4754be0f5ddf159c23fe P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:44.681114 11145 tablet_bootstrap.cc:492] T dfa48440020d4754be0f5ddf159c23fe P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:44.681231 11145 ts_tablet_manager.cc:1367] T dfa48440020d4754be0f5ddf159c23fe P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:44.681661 11145 raft_consensus.cc:348] T dfa48440020d4754be0f5ddf159c23fe P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:44.681746 11145 raft_consensus.cc:374] T dfa48440020d4754be0f5ddf159c23fe P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:44.681769 11145 raft_consensus.cc:729] T dfa48440020d4754be0f5ddf159c23fe P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:44.681888 11145 consensus_queue.cc:260] T dfa48440020d4754be0f5ddf159c23fe P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:44.682168 11145 ts_tablet_manager.cc:1398] T dfa48440020d4754be0f5ddf159c23fe P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:44.695982 10904 raft_consensus.cc:482] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:44.696087 10904 raft_consensus.cc:504] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:44.696617 10904 leader_election.cc:288] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:44.696877 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa48440020d4754be0f5ddf159c23fe" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:44.697019 9081 raft_consensus.cc:2454] T dfa48440020d4754be0f5ddf159c23fe P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:44.697085 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa48440020d4754be0f5ddf159c23fe" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:44.697240 6658 raft_consensus.cc:2454] T dfa48440020d4754be0f5ddf159c23fe P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:44.697312 6857 leader_election.cc:302] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:44.697477 10904 raft_consensus.cc:2781] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:44.697544 10904 raft_consensus.cc:482] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:44.697594 10904 raft_consensus.cc:3037] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:44.698532 10904 raft_consensus.cc:504] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:44.698865 10904 leader_election.cc:288] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:44.699158 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa48440020d4754be0f5ddf159c23fe" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:44.699216 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa48440020d4754be0f5ddf159c23fe" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:44.699263 6658 raft_consensus.cc:3037] T dfa48440020d4754be0f5ddf159c23fe P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:44.699384 9081 raft_consensus.cc:3037] T dfa48440020d4754be0f5ddf159c23fe P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:44.700277 6658 raft_consensus.cc:2454] T dfa48440020d4754be0f5ddf159c23fe P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:44.700567 9081 raft_consensus.cc:2454] T dfa48440020d4754be0f5ddf159c23fe P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:44.700855 6858 leader_election.cc:302] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:44.701073 10904 raft_consensus.cc:2781] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:44.701215 10904 raft_consensus.cc:686] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:44.701357 10904 consensus_queue.cc:237] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:44.703601 6380 catalog_manager.cc:5484] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:44.754508 10928 raft_consensus.cc:482] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:44.754652 10928 raft_consensus.cc:504] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:44.755090 10928 leader_election.cc:288] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:44.755416 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4748749ca2e94c83a9cff7e2564ba32a" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:44.755553 6528 raft_consensus.cc:2454] T 4748749ca2e94c83a9cff7e2564ba32a P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:44.755841 6595 leader_election.cc:302] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:44.756353 10928 raft_consensus.cc:2781] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:44.756407 10928 raft_consensus.cc:482] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:44.756443 10928 raft_consensus.cc:3037] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:44.757042 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4748749ca2e94c83a9cff7e2564ba32a" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:44.757169 6791 raft_consensus.cc:2454] T 4748749ca2e94c83a9cff7e2564ba32a P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 0.
I20240503 13:59:44.757614 10928 raft_consensus.cc:504] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:44.757915 10928 leader_election.cc:288] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:44.758157 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4748749ca2e94c83a9cff7e2564ba32a" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:44.758270 6527 raft_consensus.cc:3037] T 4748749ca2e94c83a9cff7e2564ba32a P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:44.759069 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4748749ca2e94c83a9cff7e2564ba32a" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:44.759193 6791 raft_consensus.cc:3037] T 4748749ca2e94c83a9cff7e2564ba32a P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:44.759224 6527 raft_consensus.cc:2454] T 4748749ca2e94c83a9cff7e2564ba32a P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:44.759503 6595 leader_election.cc:302] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:44.759671 10928 raft_consensus.cc:2781] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:44.759742 10928 raft_consensus.cc:686] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 13:59:44.759891 10928 consensus_queue.cc:237] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:44.760116 6791 raft_consensus.cc:2454] T 4748749ca2e94c83a9cff7e2564ba32a P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 13:59:44.761873 6380 catalog_manager.cc:5484] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:44.791443 10904 raft_consensus.cc:482] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:44.791579 10904 raft_consensus.cc:504] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:44.792135 10904 leader_election.cc:288] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:44.792341 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0e74cfbe1a4fbca46c100f3d0d5f8b" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:44.792368 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0e74cfbe1a4fbca46c100f3d0d5f8b" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:44.792484 9343 raft_consensus.cc:2454] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:44.792536 8947 raft_consensus.cc:2454] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:44.792860 6857 leader_election.cc:302] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:44.793026 10904 raft_consensus.cc:2781] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:44.793087 10904 raft_consensus.cc:482] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:44.793140 10904 raft_consensus.cc:3037] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:44.793967 10904 raft_consensus.cc:504] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:44.794277 10904 leader_election.cc:288] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:44.794591 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0e74cfbe1a4fbca46c100f3d0d5f8b" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:44.794591 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0e74cfbe1a4fbca46c100f3d0d5f8b" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:44.794711 8947 raft_consensus.cc:3037] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:44.794711 9343 raft_consensus.cc:3037] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:44.795689 8947 raft_consensus.cc:2454] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:44.795689 9343 raft_consensus.cc:2454] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:44.795990 6857 leader_election.cc:302] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:44.796175 10904 raft_consensus.cc:2781] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:44.796257 10904 raft_consensus.cc:686] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:44.796383 10904 consensus_queue.cc:237] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:44.798700 6380 catalog_manager.cc:5484] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:44.922828 10932 raft_consensus.cc:482] T e4fce5deaae4457a9f161e66fcc02c60 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:44.922968 10932 raft_consensus.cc:504] T e4fce5deaae4457a9f161e66fcc02c60 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:44.923413 10932 leader_election.cc:288] T e4fce5deaae4457a9f161e66fcc02c60 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:44.923835 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4fce5deaae4457a9f161e66fcc02c60" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:44.923904 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4fce5deaae4457a9f161e66fcc02c60" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:44.923972 9205 raft_consensus.cc:2454] T e4fce5deaae4457a9f161e66fcc02c60 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:44.924052 6922 raft_consensus.cc:2454] T e4fce5deaae4457a9f161e66fcc02c60 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 0.
I20240503 13:59:44.924314 9016 leader_election.cc:302] T e4fce5deaae4457a9f161e66fcc02c60 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:44.924566 10932 raft_consensus.cc:2781] T e4fce5deaae4457a9f161e66fcc02c60 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:44.924615 10932 raft_consensus.cc:482] T e4fce5deaae4457a9f161e66fcc02c60 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:44.924660 10932 raft_consensus.cc:3037] T e4fce5deaae4457a9f161e66fcc02c60 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:44.925601 10932 raft_consensus.cc:504] T e4fce5deaae4457a9f161e66fcc02c60 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:44.925941 10932 leader_election.cc:288] T e4fce5deaae4457a9f161e66fcc02c60 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:44.926302 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4fce5deaae4457a9f161e66fcc02c60" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:44.926295 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4fce5deaae4457a9f161e66fcc02c60" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:44.926425 6922 raft_consensus.cc:3037] T e4fce5deaae4457a9f161e66fcc02c60 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:44.926424 9205 raft_consensus.cc:3037] T e4fce5deaae4457a9f161e66fcc02c60 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:44.927421 9205 raft_consensus.cc:2454] T e4fce5deaae4457a9f161e66fcc02c60 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:44.927582 6922 raft_consensus.cc:2454] T e4fce5deaae4457a9f161e66fcc02c60 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 13:59:44.927834 9016 leader_election.cc:302] T e4fce5deaae4457a9f161e66fcc02c60 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:44.928105 10932 raft_consensus.cc:2781] T e4fce5deaae4457a9f161e66fcc02c60 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:44.928275 10932 raft_consensus.cc:686] T e4fce5deaae4457a9f161e66fcc02c60 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 13:59:44.928402 10932 consensus_queue.cc:237] T e4fce5deaae4457a9f161e66fcc02c60 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:44.930605 6380 catalog_manager.cc:5484] T e4fce5deaae4457a9f161e66fcc02c60 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:44.956488 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:44.960692 9343 raft_consensus.cc:1261] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:44.960692 8947 raft_consensus.cc:1261] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:44.961086 11116 consensus_queue.cc:1035] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:44.961494 11116 consensus_queue.cc:1035] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:44.968492 6380 catalog_manager.cc:5576] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931) does not have the latest schema for tablet ed0e74cfbe1a4fbca46c100f3d0d5f8b (table rebalancer_dynamic_table [id=1283c7de26ed4f74b345e29414b410a8]). Expected version 1 got 0
I20240503 13:59:44.968878 11163 tablet.cc:1722] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:44.969579 11165 tablet.cc:1722] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:44.969939 11165 tablet.cc:1620] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:44.970089 6385 catalog_manager.cc:5576] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531) does not have the latest schema for tablet ed0e74cfbe1a4fbca46c100f3d0d5f8b (table rebalancer_dynamic_table [id=1283c7de26ed4f74b345e29414b410a8]). Expected version 1 got 0
I20240503 13:59:44.972204 11164 tablet.cc:1722] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:44.972409 6385 catalog_manager.cc:5576] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547) does not have the latest schema for tablet ed0e74cfbe1a4fbca46c100f3d0d5f8b (table rebalancer_dynamic_table [id=1283c7de26ed4f74b345e29414b410a8]). Expected version 1 got 0
I20240503 13:59:44.972544 11164 tablet.cc:1620] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:44.975978 6385 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=1283c7de26ed4f74b345e29414b410a8] alter complete (version 1)
I20240503 13:59:44.972035 11160 mvcc.cc:205] Tried to move back new op lower bound from 7023594639195140096 to 7023594638527266816. Current Snapshot: MvccSnapshot[applied={T|T < 7023594639195140096}]
I20240503 13:59:44.978976 11163 tablet.cc:1620] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460: MemRowSet was empty: no flush needed.
I20240503 13:59:45.080397 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:45.085914 11163 tablet.cc:1722] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:45.086272 11163 tablet.cc:1620] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460: MemRowSet was empty: no flush needed.
I20240503 13:59:45.087250 11164 tablet.cc:1722] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:45.087055 11165 tablet.cc:1722] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:45.087678 11164 tablet.cc:1620] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 06ca8dbc004c42b99faa527a4e6f9545: MemRowSet was empty: no flush needed.
I20240503 13:59:45.087872 11165 tablet.cc:1620] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:45.092182 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=1283c7de26ed4f74b345e29414b410a8] alter complete (version 2)
I20240503 13:59:45.096771 11116 consensus_queue.cc:1035] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:45.101138 11116 consensus_queue.cc:1035] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:45.193262 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:45.198798 11163 tablet.cc:1722] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:45.199471 11165 tablet.cc:1722] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:45.199482 11164 tablet.cc:1722] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 06ca8dbc004c42b99faa527a4e6f9545: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:45.204897 6385 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=1283c7de26ed4f74b345e29414b410a8] alter complete (version 3)
I20240503 13:59:45.308463 6379 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:45.308630 6379 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:45.310657 6379 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet ed0e74cfbe1a4fbca46c100f3d0d5f8b
I20240503 13:59:45.311457 8926 tablet_service.cc:1514] Processing DeleteTablet for tablet ed0e74cfbe1a4fbca46c100f3d0d5f8b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:45 UTC) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:45.311615 9321 tablet_service.cc:1514] Processing DeleteTablet for tablet ed0e74cfbe1a4fbca46c100f3d0d5f8b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:45 UTC) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:45.311830 11173 tablet_replica.cc:330] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:45.312022 6901 tablet_service.cc:1514] Processing DeleteTablet for tablet ed0e74cfbe1a4fbca46c100f3d0d5f8b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:45 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:45.312106 11173 raft_consensus.cc:2229] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:45.312250 11173 raft_consensus.cc:2258] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:45.312366 11174 tablet_replica.cc:330] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:45.312556 11174 raft_consensus.cc:2229] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:45.312712 11174 raft_consensus.cc:2258] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:45.313283 11174 ts_tablet_manager.cc:1875] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:45.313570 11173 ts_tablet_manager.cc:1875] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:45.315565 11174 ts_tablet_manager.cc:1888] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:45.315667 11174 log.cc:1196] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/ed0e74cfbe1a4fbca46c100f3d0d5f8b
I20240503 13:59:45.315982 11174 ts_tablet_manager.cc:1909] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 0d77752181e944698e809e325e5937e6: Deleting consensus metadata
I20240503 13:59:45.316409 11173 ts_tablet_manager.cc:1888] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:45.316511 11173 log.cc:1196] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/ed0e74cfbe1a4fbca46c100f3d0d5f8b
I20240503 13:59:45.316650 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet ed0e74cfbe1a4fbca46c100f3d0d5f8b (table rebalancer_dynamic_table_new_name [id=1283c7de26ed4f74b345e29414b410a8]) successfully deleted
I20240503 13:59:45.316861 11173 ts_tablet_manager.cc:1909] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P 06ca8dbc004c42b99faa527a4e6f9545: Deleting consensus metadata
I20240503 13:59:45.317148 11175 tablet_replica.cc:330] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:45.317320 11175 raft_consensus.cc:2229] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:45.317520 11175 raft_consensus.cc:2258] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:45.317538 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet ed0e74cfbe1a4fbca46c100f3d0d5f8b (table rebalancer_dynamic_table_new_name [id=1283c7de26ed4f74b345e29414b410a8]) successfully deleted
I20240503 13:59:45.323668 11175 ts_tablet_manager.cc:1875] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:45.325807 11175 ts_tablet_manager.cc:1888] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:45.325901 11175 log.cc:1196] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/ed0e74cfbe1a4fbca46c100f3d0d5f8b
I20240503 13:59:45.326300 11175 ts_tablet_manager.cc:1909] T ed0e74cfbe1a4fbca46c100f3d0d5f8b P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:45.327075 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet ed0e74cfbe1a4fbca46c100f3d0d5f8b (table rebalancer_dynamic_table_new_name [id=1283c7de26ed4f74b345e29414b410a8]) successfully deleted
I20240503 13:59:45.339469 11120 consensus_queue.cc:1035] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:45.343928 11177 consensus_queue.cc:1035] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:45.382957 11105 consensus_queue.cc:1035] T e4fce5deaae4457a9f161e66fcc02c60 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:45.386783 11159 consensus_queue.cc:1035] T e4fce5deaae4457a9f161e66fcc02c60 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:45.412741 6379 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:45.418327 9321 tablet_service.cc:1467] Processing CreateTablet for tablet d3ebef1cb4264054998617950c35508a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3cda36e1926842f2a341560c8f7a2a6e]), partition=
I20240503 13:59:45.418439 6507 tablet_service.cc:1467] Processing CreateTablet for tablet d3ebef1cb4264054998617950c35508a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3cda36e1926842f2a341560c8f7a2a6e]), partition=
I20240503 13:59:45.418632 9321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3ebef1cb4264054998617950c35508a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:45.418705 6507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3ebef1cb4264054998617950c35508a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:45.420529 11186 tablet_bootstrap.cc:492] T d3ebef1cb4264054998617950c35508a P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:45.420712 6901 tablet_service.cc:1467] Processing CreateTablet for tablet d3ebef1cb4264054998617950c35508a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3cda36e1926842f2a341560c8f7a2a6e]), partition=
I20240503 13:59:45.420926 11185 tablet_bootstrap.cc:492] T d3ebef1cb4264054998617950c35508a P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:45.420954 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3ebef1cb4264054998617950c35508a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:45.421741 11186 tablet_bootstrap.cc:654] T d3ebef1cb4264054998617950c35508a P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:45.422500 11185 tablet_bootstrap.cc:654] T d3ebef1cb4264054998617950c35508a P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:45.422829 11186 tablet_bootstrap.cc:492] T d3ebef1cb4264054998617950c35508a P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:45.422889 11186 ts_tablet_manager.cc:1367] T d3ebef1cb4264054998617950c35508a P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:45.423350 11186 raft_consensus.cc:348] T d3ebef1cb4264054998617950c35508a P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:45.423568 11186 raft_consensus.cc:374] T d3ebef1cb4264054998617950c35508a P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:45.423601 11186 raft_consensus.cc:729] T d3ebef1cb4264054998617950c35508a P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:45.423604 11185 tablet_bootstrap.cc:492] T d3ebef1cb4264054998617950c35508a P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:45.423688 11185 ts_tablet_manager.cc:1367] T d3ebef1cb4264054998617950c35508a P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:45.423715 11186 consensus_queue.cc:260] T d3ebef1cb4264054998617950c35508a P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:45.424163 11185 raft_consensus.cc:348] T d3ebef1cb4264054998617950c35508a P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:45.424261 11185 raft_consensus.cc:374] T d3ebef1cb4264054998617950c35508a P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:45.424291 11185 raft_consensus.cc:729] T d3ebef1cb4264054998617950c35508a P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:45.424806 11186 ts_tablet_manager.cc:1398] T d3ebef1cb4264054998617950c35508a P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:45.424408 11185 consensus_queue.cc:260] T d3ebef1cb4264054998617950c35508a P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:45.425575 11185 ts_tablet_manager.cc:1398] T d3ebef1cb4264054998617950c35508a P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:45.425884 11189 tablet_bootstrap.cc:492] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:45.427487 11189 tablet_bootstrap.cc:654] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:45.428668 11189 tablet_bootstrap.cc:492] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:45.428797 11189 ts_tablet_manager.cc:1367] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:45.429306 11189 raft_consensus.cc:348] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:45.429420 11189 raft_consensus.cc:374] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:45.429452 11189 raft_consensus.cc:729] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:45.429620 11189 consensus_queue.cc:260] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:45.430027 11189 ts_tablet_manager.cc:1398] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:45.456492 6380 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:45.462785 6901 tablet_service.cc:1467] Processing CreateTablet for tablet 442c955ab10f4ecbafdc9bf89d8fb39e (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=7af8efee9ffd469b98e12ab7f8005bbc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:45.462992 9322 tablet_service.cc:1467] Processing CreateTablet for tablet abe859639139483aa1e57ee6c9722b87 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=7af8efee9ffd469b98e12ab7f8005bbc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:45.462985 9190 tablet_service.cc:1467] Processing CreateTablet for tablet abe859639139483aa1e57ee6c9722b87 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=7af8efee9ffd469b98e12ab7f8005bbc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:45.463099 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442c955ab10f4ecbafdc9bf89d8fb39e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:45.463263 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abe859639139483aa1e57ee6c9722b87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:45.463261 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abe859639139483aa1e57ee6c9722b87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:45.463537 9321 tablet_service.cc:1467] Processing CreateTablet for tablet 442c955ab10f4ecbafdc9bf89d8fb39e (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=7af8efee9ffd469b98e12ab7f8005bbc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:45.463649 6769 tablet_service.cc:1467] Processing CreateTablet for tablet 442c955ab10f4ecbafdc9bf89d8fb39e (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=7af8efee9ffd469b98e12ab7f8005bbc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:45.463778 9321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442c955ab10f4ecbafdc9bf89d8fb39e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:45.463903 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442c955ab10f4ecbafdc9bf89d8fb39e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:45.466197 6771 tablet_service.cc:1467] Processing CreateTablet for tablet 0a719fd125e54a0fa1c9b42a92548da0 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=7af8efee9ffd469b98e12ab7f8005bbc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:45.466209 8926 tablet_service.cc:1467] Processing CreateTablet for tablet 0a719fd125e54a0fa1c9b42a92548da0 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=7af8efee9ffd469b98e12ab7f8005bbc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:45.466421 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a719fd125e54a0fa1c9b42a92548da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:45.466513 8926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a719fd125e54a0fa1c9b42a92548da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:45.466744 11185 tablet_bootstrap.cc:492] T abe859639139483aa1e57ee6c9722b87 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:45.467782 11185 tablet_bootstrap.cc:654] T abe859639139483aa1e57ee6c9722b87 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:45.468582 11192 tablet_bootstrap.cc:492] T 442c955ab10f4ecbafdc9bf89d8fb39e P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:45.469686 11192 tablet_bootstrap.cc:654] T 442c955ab10f4ecbafdc9bf89d8fb39e P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:45.469695 6507 tablet_service.cc:1467] Processing CreateTablet for tablet abe859639139483aa1e57ee6c9722b87 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=7af8efee9ffd469b98e12ab7f8005bbc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:45.469862 11189 tablet_bootstrap.cc:492] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:45.469975 6507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abe859639139483aa1e57ee6c9722b87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:45.470896 11189 tablet_bootstrap.cc:654] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:45.471709 11194 tablet_bootstrap.cc:492] T 0a719fd125e54a0fa1c9b42a92548da0 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:45.471999 11186 tablet_bootstrap.cc:492] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:45.472630 11191 tablet_bootstrap.cc:492] T abe859639139483aa1e57ee6c9722b87 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:45.473042 11186 tablet_bootstrap.cc:654] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:45.473387 11189 tablet_bootstrap.cc:492] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:45.473470 11189 ts_tablet_manager.cc:1367] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:45.474323 9060 tablet_service.cc:1467] Processing CreateTablet for tablet 0a719fd125e54a0fa1c9b42a92548da0 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=7af8efee9ffd469b98e12ab7f8005bbc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:45.474599 9060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a719fd125e54a0fa1c9b42a92548da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:45.474617 11189 raft_consensus.cc:348] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:45.474721 11189 raft_consensus.cc:374] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:45.474750 11189 raft_consensus.cc:729] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:45.474999 11189 consensus_queue.cc:260] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:45.475342 11189 ts_tablet_manager.cc:1398] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:45.475522 11194 tablet_bootstrap.cc:654] T 0a719fd125e54a0fa1c9b42a92548da0 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:45.476792 11186 tablet_bootstrap.cc:492] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:45.476882 11186 ts_tablet_manager.cc:1367] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:45.477007 11194 tablet_bootstrap.cc:492] T 0a719fd125e54a0fa1c9b42a92548da0 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:45.477116 11194 ts_tablet_manager.cc:1367] T 0a719fd125e54a0fa1c9b42a92548da0 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:59:45.477360 11186 raft_consensus.cc:348] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:45.477465 11186 raft_consensus.cc:374] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:45.477501 11186 raft_consensus.cc:729] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:45.477520 11194 raft_consensus.cc:348] T 0a719fd125e54a0fa1c9b42a92548da0 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:45.477627 11194 raft_consensus.cc:374] T 0a719fd125e54a0fa1c9b42a92548da0 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:45.477665 11194 raft_consensus.cc:729] T 0a719fd125e54a0fa1c9b42a92548da0 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:45.477649 11186 consensus_queue.cc:260] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:45.477725 11191 tablet_bootstrap.cc:654] T abe859639139483aa1e57ee6c9722b87 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:45.477794 11194 consensus_queue.cc:260] T 0a719fd125e54a0fa1c9b42a92548da0 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:45.478169 11194 ts_tablet_manager.cc:1398] T 0a719fd125e54a0fa1c9b42a92548da0 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:45.479637 11186 ts_tablet_manager.cc:1398] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:45.481047 11191 tablet_bootstrap.cc:492] T abe859639139483aa1e57ee6c9722b87 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:45.481143 11191 ts_tablet_manager.cc:1367] T abe859639139483aa1e57ee6c9722b87 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:59:45.481577 11191 raft_consensus.cc:348] T abe859639139483aa1e57ee6c9722b87 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:45.482168 11191 raft_consensus.cc:374] T abe859639139483aa1e57ee6c9722b87 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:45.482251 11191 raft_consensus.cc:729] T abe859639139483aa1e57ee6c9722b87 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:45.482421 11191 consensus_queue.cc:260] T abe859639139483aa1e57ee6c9722b87 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:45.482671 11199 tablet_bootstrap.cc:492] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:45.485493 11199 tablet_bootstrap.cc:654] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:45.486675 11191 ts_tablet_manager.cc:1398] T abe859639139483aa1e57ee6c9722b87 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:59:45.486781 11185 tablet_bootstrap.cc:492] T abe859639139483aa1e57ee6c9722b87 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:45.486891 11185 ts_tablet_manager.cc:1367] T abe859639139483aa1e57ee6c9722b87 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.002s
I20240503 13:59:45.487432 11185 raft_consensus.cc:348] T abe859639139483aa1e57ee6c9722b87 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:45.487560 11185 raft_consensus.cc:374] T abe859639139483aa1e57ee6c9722b87 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:45.487598 11185 raft_consensus.cc:729] T abe859639139483aa1e57ee6c9722b87 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:45.487743 11185 consensus_queue.cc:260] T abe859639139483aa1e57ee6c9722b87 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:45.488063 11185 ts_tablet_manager.cc:1398] T abe859639139483aa1e57ee6c9722b87 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 13:59:45.488258 11185 tablet_bootstrap.cc:492] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:45.489280 11185 tablet_bootstrap.cc:654] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:45.490823 11192 tablet_bootstrap.cc:492] T 442c955ab10f4ecbafdc9bf89d8fb39e P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:45.490926 11192 ts_tablet_manager.cc:1367] T 442c955ab10f4ecbafdc9bf89d8fb39e P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240503 13:59:45.491027 11199 tablet_bootstrap.cc:492] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:45.491118 11199 ts_tablet_manager.cc:1367] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:59:45.491402 11192 raft_consensus.cc:348] T 442c955ab10f4ecbafdc9bf89d8fb39e P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:45.491506 11192 raft_consensus.cc:374] T 442c955ab10f4ecbafdc9bf89d8fb39e P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:45.491541 11192 raft_consensus.cc:729] T 442c955ab10f4ecbafdc9bf89d8fb39e P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:45.491658 11192 consensus_queue.cc:260] T 442c955ab10f4ecbafdc9bf89d8fb39e P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:45.491636 11199 raft_consensus.cc:348] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:45.491748 11199 raft_consensus.cc:374] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:45.491953 11192 ts_tablet_manager.cc:1398] T 442c955ab10f4ecbafdc9bf89d8fb39e P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:45.492098 11192 tablet_bootstrap.cc:492] T 0a719fd125e54a0fa1c9b42a92548da0 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:45.491909 11199 raft_consensus.cc:729] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:45.492270 11199 consensus_queue.cc:260] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:45.492638 11199 ts_tablet_manager.cc:1398] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:45.492749 11185 tablet_bootstrap.cc:492] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:45.492821 11185 ts_tablet_manager.cc:1367] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240503 13:59:45.493031 11192 tablet_bootstrap.cc:654] T 0a719fd125e54a0fa1c9b42a92548da0 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:45.493151 11185 raft_consensus.cc:348] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:45.493222 11185 raft_consensus.cc:374] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:45.493243 11185 raft_consensus.cc:729] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:45.493342 11185 consensus_queue.cc:260] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:45.493621 11185 ts_tablet_manager.cc:1398] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:45.503401 11192 tablet_bootstrap.cc:492] T 0a719fd125e54a0fa1c9b42a92548da0 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:45.503505 11192 ts_tablet_manager.cc:1367] T 0a719fd125e54a0fa1c9b42a92548da0 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 13:59:45.504019 11192 raft_consensus.cc:348] T 0a719fd125e54a0fa1c9b42a92548da0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:45.504141 11192 raft_consensus.cc:374] T 0a719fd125e54a0fa1c9b42a92548da0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:45.504173 11192 raft_consensus.cc:729] T 0a719fd125e54a0fa1c9b42a92548da0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:45.504293 11192 consensus_queue.cc:260] T 0a719fd125e54a0fa1c9b42a92548da0 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:45.504941 11192 ts_tablet_manager.cc:1398] T 0a719fd125e54a0fa1c9b42a92548da0 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:45.510371 11172 raft_consensus.cc:482] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:45.510493 11172 raft_consensus.cc:504] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:45.510917 11172 leader_election.cc:288] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:45.511191 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442c955ab10f4ecbafdc9bf89d8fb39e" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:45.511360 6922 raft_consensus.cc:2454] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:45.511353 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442c955ab10f4ecbafdc9bf89d8fb39e" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:45.511476 6791 raft_consensus.cc:2454] T 442c955ab10f4ecbafdc9bf89d8fb39e P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:45.511713 9278 leader_election.cc:302] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:45.512647 11172 raft_consensus.cc:2781] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:45.512712 11172 raft_consensus.cc:482] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:45.512753 11172 raft_consensus.cc:3037] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:45.513891 11172 raft_consensus.cc:504] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:45.514184 11172 leader_election.cc:288] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:45.514600 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442c955ab10f4ecbafdc9bf89d8fb39e" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:45.514616 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442c955ab10f4ecbafdc9bf89d8fb39e" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:45.514714 6922 raft_consensus.cc:3037] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:45.514740 6791 raft_consensus.cc:3037] T 442c955ab10f4ecbafdc9bf89d8fb39e P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:45.515614 6791 raft_consensus.cc:2454] T 442c955ab10f4ecbafdc9bf89d8fb39e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:45.515625 6922 raft_consensus.cc:2454] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:45.515875 9277 leader_election.cc:302] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:45.516062 11172 raft_consensus.cc:2781] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:45.516151 11172 raft_consensus.cc:686] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:45.516307 11172 consensus_queue.cc:237] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:45.519727 6383 catalog_manager.cc:5484] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:45.666694 11116 raft_consensus.cc:482] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:45.666819 11116 raft_consensus.cc:504] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:45.667266 11116 leader_election.cc:288] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:45.667640 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ebef1cb4264054998617950c35508a" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:45.667646 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ebef1cb4264054998617950c35508a" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:45.667780 6527 raft_consensus.cc:2454] T d3ebef1cb4264054998617950c35508a P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:45.667781 9343 raft_consensus.cc:2454] T d3ebef1cb4264054998617950c35508a P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:45.668064 6858 leader_election.cc:302] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:45.668215 11116 raft_consensus.cc:2781] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:45.668277 11116 raft_consensus.cc:482] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:45.668315 11116 raft_consensus.cc:3037] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:45.669301 11116 raft_consensus.cc:504] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:45.669651 11116 leader_election.cc:288] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:45.669982 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ebef1cb4264054998617950c35508a" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:45.669996 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ebef1cb4264054998617950c35508a" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:45.670105 9343 raft_consensus.cc:3037] T d3ebef1cb4264054998617950c35508a P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:45.670094 6527 raft_consensus.cc:3037] T d3ebef1cb4264054998617950c35508a P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:45.671161 6527 raft_consensus.cc:2454] T d3ebef1cb4264054998617950c35508a P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:45.671180 9343 raft_consensus.cc:2454] T d3ebef1cb4264054998617950c35508a P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:45.671463 6858 leader_election.cc:302] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:45.671627 11116 raft_consensus.cc:2781] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:45.671717 11116 raft_consensus.cc:686] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:45.671851 11116 consensus_queue.cc:237] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:45.674571 6383 catalog_manager.cc:5484] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:45.784377 11205 raft_consensus.cc:482] T 0a719fd125e54a0fa1c9b42a92548da0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:45.784509 11205 raft_consensus.cc:504] T 0a719fd125e54a0fa1c9b42a92548da0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:45.784981 11205 leader_election.cc:288] T 0a719fd125e54a0fa1c9b42a92548da0 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:45.785396 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a719fd125e54a0fa1c9b42a92548da0" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:45.785398 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a719fd125e54a0fa1c9b42a92548da0" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:45.785543 8948 raft_consensus.cc:2454] T 0a719fd125e54a0fa1c9b42a92548da0 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:45.785560 9080 raft_consensus.cc:2454] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:45.785825 6726 leader_election.cc:302] T 0a719fd125e54a0fa1c9b42a92548da0 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:45.785998 11205 raft_consensus.cc:2781] T 0a719fd125e54a0fa1c9b42a92548da0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:45.786058 11205 raft_consensus.cc:482] T 0a719fd125e54a0fa1c9b42a92548da0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:45.786093 11205 raft_consensus.cc:3037] T 0a719fd125e54a0fa1c9b42a92548da0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:45.787632 11205 raft_consensus.cc:504] T 0a719fd125e54a0fa1c9b42a92548da0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:45.787971 11205 leader_election.cc:288] T 0a719fd125e54a0fa1c9b42a92548da0 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:45.788302 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a719fd125e54a0fa1c9b42a92548da0" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:45.788318 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a719fd125e54a0fa1c9b42a92548da0" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:45.788421 8948 raft_consensus.cc:3037] T 0a719fd125e54a0fa1c9b42a92548da0 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:45.788429 9080 raft_consensus.cc:3037] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:45.789220 8948 raft_consensus.cc:2454] T 0a719fd125e54a0fa1c9b42a92548da0 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:45.789299 9080 raft_consensus.cc:2454] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:45.789487 6726 leader_election.cc:302] T 0a719fd125e54a0fa1c9b42a92548da0 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:45.789629 11205 raft_consensus.cc:2781] T 0a719fd125e54a0fa1c9b42a92548da0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:45.789731 11205 raft_consensus.cc:686] T 0a719fd125e54a0fa1c9b42a92548da0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:45.789891 11205 consensus_queue.cc:237] T 0a719fd125e54a0fa1c9b42a92548da0 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 13:59:45.792116 6383 catalog_manager.cc:5484] T 0a719fd125e54a0fa1c9b42a92548da0 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:45.824671 6383 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:45.828573 6527 raft_consensus.cc:1261] T d3ebef1cb4264054998617950c35508a P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:45.828624 9343 raft_consensus.cc:1261] T d3ebef1cb4264054998617950c35508a P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:45.829016 11166 consensus_queue.cc:1035] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:45.829299 11166 consensus_queue.cc:1035] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:45.833874 6379 catalog_manager.cc:5576] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021) does not have the latest schema for tablet d3ebef1cb4264054998617950c35508a (table rebalancer_dynamic_table [id=3cda36e1926842f2a341560c8f7a2a6e]). Expected version 1 got 0
I20240503 13:59:45.835601 11213 tablet.cc:1722] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:45.837260 6379 catalog_manager.cc:5576] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531) does not have the latest schema for tablet d3ebef1cb4264054998617950c35508a (table rebalancer_dynamic_table [id=3cda36e1926842f2a341560c8f7a2a6e]). Expected version 1 got 0
I20240503 13:59:45.837607 6383 catalog_manager.cc:5576] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931) does not have the latest schema for tablet d3ebef1cb4264054998617950c35508a (table rebalancer_dynamic_table [id=3cda36e1926842f2a341560c8f7a2a6e]). Expected version 1 got 0
I20240503 13:59:45.837702 11214 tablet.cc:1722] T d3ebef1cb4264054998617950c35508a P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:45.838253 11213 tablet.cc:1620] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460: MemRowSet was empty: no flush needed.
I20240503 13:59:45.840842 11180 mvcc.cc:205] Tried to move back new op lower bound from 7023594642750304256 to 7023594642113298432. Current Snapshot: MvccSnapshot[applied={T|T < 7023594642750304256}]
I20240503 13:59:45.842484 11215 tablet.cc:1722] T d3ebef1cb4264054998617950c35508a P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:45.842785 11215 tablet.cc:1620] T d3ebef1cb4264054998617950c35508a P 5f46e19644fd4f4b825c86ba6af768da: MemRowSet was empty: no flush needed.
I20240503 13:59:45.844861 11214 tablet.cc:1620] T d3ebef1cb4264054998617950c35508a P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:45.844640 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=3cda36e1926842f2a341560c8f7a2a6e] alter complete (version 1)
I20240503 13:59:45.907351 11211 raft_consensus.cc:482] T abe859639139483aa1e57ee6c9722b87 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:45.907472 11211 raft_consensus.cc:504] T abe859639139483aa1e57ee6c9722b87 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:45.907912 11211 leader_election.cc:288] T abe859639139483aa1e57ee6c9722b87 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:45.908164 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abe859639139483aa1e57ee6c9722b87" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:45.908247 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abe859639139483aa1e57ee6c9722b87" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:45.908298 9205 raft_consensus.cc:2454] T abe859639139483aa1e57ee6c9722b87 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:45.908404 6527 raft_consensus.cc:2454] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:45.908697 9278 leader_election.cc:302] T abe859639139483aa1e57ee6c9722b87 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:45.908877 11211 raft_consensus.cc:2781] T abe859639139483aa1e57ee6c9722b87 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:45.908994 11211 raft_consensus.cc:482] T abe859639139483aa1e57ee6c9722b87 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:45.909040 11211 raft_consensus.cc:3037] T abe859639139483aa1e57ee6c9722b87 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:45.910028 11211 raft_consensus.cc:504] T abe859639139483aa1e57ee6c9722b87 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:45.910432 11211 leader_election.cc:288] T abe859639139483aa1e57ee6c9722b87 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:45.910689 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abe859639139483aa1e57ee6c9722b87" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:45.910799 9205 raft_consensus.cc:3037] T abe859639139483aa1e57ee6c9722b87 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:45.910769 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abe859639139483aa1e57ee6c9722b87" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:45.910892 6527 raft_consensus.cc:3037] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:45.911824 9205 raft_consensus.cc:2454] T abe859639139483aa1e57ee6c9722b87 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:45.911824 6527 raft_consensus.cc:2454] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:45.912130 9278 leader_election.cc:302] T abe859639139483aa1e57ee6c9722b87 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:45.912313 11211 raft_consensus.cc:2781] T abe859639139483aa1e57ee6c9722b87 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:45.912521 11211 raft_consensus.cc:686] T abe859639139483aa1e57ee6c9722b87 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:45.912672 11211 consensus_queue.cc:237] T abe859639139483aa1e57ee6c9722b87 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:45.915048 6385 catalog_manager.cc:5484] T abe859639139483aa1e57ee6c9722b87 P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:45.949712 6383 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:45.955559 11213 tablet.cc:1722] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:45.955983 11215 tablet.cc:1722] T d3ebef1cb4264054998617950c35508a P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:45.956079 11213 tablet.cc:1620] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460: MemRowSet was empty: no flush needed.
I20240503 13:59:45.956293 11215 tablet.cc:1620] T d3ebef1cb4264054998617950c35508a P 5f46e19644fd4f4b825c86ba6af768da: MemRowSet was empty: no flush needed.
I20240503 13:59:45.956946 11214 tablet.cc:1722] T d3ebef1cb4264054998617950c35508a P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:45.957217 11214 tablet.cc:1620] T d3ebef1cb4264054998617950c35508a P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:45.962671 6383 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=3cda36e1926842f2a341560c8f7a2a6e] alter complete (version 2)
I20240503 13:59:45.969655 11211 consensus_queue.cc:1035] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:45.972589 11167 consensus_queue.cc:1035] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:46.065001 6385 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:46.070855 11213 tablet.cc:1722] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:46.071175 11214 tablet.cc:1722] T d3ebef1cb4264054998617950c35508a P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:46.071784 11215 tablet.cc:1722] T d3ebef1cb4264054998617950c35508a P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:46.077201 6385 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=3cda36e1926842f2a341560c8f7a2a6e] alter complete (version 3)
I20240503 13:59:46.181090 6380 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:46.181259 6380 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:46.183393 6380 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet d3ebef1cb4264054998617950c35508a
I20240503 13:59:46.183929 6901 tablet_service.cc:1514] Processing DeleteTablet for tablet d3ebef1cb4264054998617950c35508a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:46 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:46.184016 9321 tablet_service.cc:1514] Processing DeleteTablet for tablet d3ebef1cb4264054998617950c35508a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:46 UTC) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:46.184307 11223 tablet_replica.cc:330] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:46.184410 6507 tablet_service.cc:1514] Processing DeleteTablet for tablet d3ebef1cb4264054998617950c35508a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:46 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:46.184458 11224 tablet_replica.cc:330] T d3ebef1cb4264054998617950c35508a P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:46.184494 11223 raft_consensus.cc:2229] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:46.184612 11224 raft_consensus.cc:2229] T d3ebef1cb4264054998617950c35508a P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:46.184718 11225 tablet_replica.cc:330] T d3ebef1cb4264054998617950c35508a P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:46.184737 11224 raft_consensus.cc:2258] T d3ebef1cb4264054998617950c35508a P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:46.184727 11223 raft_consensus.cc:2258] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:46.185300 11223 ts_tablet_manager.cc:1875] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:46.185336 11224 ts_tablet_manager.cc:1875] T d3ebef1cb4264054998617950c35508a P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:46.187486 11225 raft_consensus.cc:2229] T d3ebef1cb4264054998617950c35508a P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:46.187606 11223 ts_tablet_manager.cc:1888] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:46.187609 11224 ts_tablet_manager.cc:1888] T d3ebef1cb4264054998617950c35508a P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:46.187644 11225 raft_consensus.cc:2258] T d3ebef1cb4264054998617950c35508a P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:46.187681 11223 log.cc:1196] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/d3ebef1cb4264054998617950c35508a
I20240503 13:59:46.187685 11224 log.cc:1196] T d3ebef1cb4264054998617950c35508a P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/d3ebef1cb4264054998617950c35508a
I20240503 13:59:46.187930 11224 ts_tablet_manager.cc:1909] T d3ebef1cb4264054998617950c35508a P 0d77752181e944698e809e325e5937e6: Deleting consensus metadata
I20240503 13:59:46.188004 11223 ts_tablet_manager.cc:1909] T d3ebef1cb4264054998617950c35508a P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:46.188335 11225 ts_tablet_manager.cc:1875] T d3ebef1cb4264054998617950c35508a P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:46.188700 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet d3ebef1cb4264054998617950c35508a (table rebalancer_dynamic_table_new_name [id=3cda36e1926842f2a341560c8f7a2a6e]) successfully deleted
I20240503 13:59:46.188704 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet d3ebef1cb4264054998617950c35508a (table rebalancer_dynamic_table_new_name [id=3cda36e1926842f2a341560c8f7a2a6e]) successfully deleted
I20240503 13:59:46.190358 11225 ts_tablet_manager.cc:1888] T d3ebef1cb4264054998617950c35508a P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:46.190423 11225 log.cc:1196] T d3ebef1cb4264054998617950c35508a P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/d3ebef1cb4264054998617950c35508a
I20240503 13:59:46.190662 11225 ts_tablet_manager.cc:1909] T d3ebef1cb4264054998617950c35508a P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:46.191320 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet d3ebef1cb4264054998617950c35508a (table rebalancer_dynamic_table_new_name [id=3cda36e1926842f2a341560c8f7a2a6e]) successfully deleted
I20240503 13:59:46.197962 11205 consensus_queue.cc:1035] T 0a719fd125e54a0fa1c9b42a92548da0 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:46.200866 11227 consensus_queue.cc:1035] T 0a719fd125e54a0fa1c9b42a92548da0 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:46.284945 6385 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:46.290793 9190 tablet_service.cc:1467] Processing CreateTablet for tablet 738b8eadbc164d43bae145740a520d7c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1f620bc625904450aaf239a47a874516]), partition=
I20240503 13:59:46.291100 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 738b8eadbc164d43bae145740a520d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:46.291128 6639 tablet_service.cc:1467] Processing CreateTablet for tablet 738b8eadbc164d43bae145740a520d7c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1f620bc625904450aaf239a47a874516]), partition=
I20240503 13:59:46.291390 6639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 738b8eadbc164d43bae145740a520d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:46.293365 11231 tablet_bootstrap.cc:492] T 738b8eadbc164d43bae145740a520d7c P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:46.293838 11232 tablet_bootstrap.cc:492] T 738b8eadbc164d43bae145740a520d7c P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:46.294576 11231 tablet_bootstrap.cc:654] T 738b8eadbc164d43bae145740a520d7c P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:46.295046 11232 tablet_bootstrap.cc:654] T 738b8eadbc164d43bae145740a520d7c P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:46.295385 6507 tablet_service.cc:1467] Processing CreateTablet for tablet 738b8eadbc164d43bae145740a520d7c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1f620bc625904450aaf239a47a874516]), partition=
I20240503 13:59:46.295644 6507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 738b8eadbc164d43bae145740a520d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:46.296401 11232 tablet_bootstrap.cc:492] T 738b8eadbc164d43bae145740a520d7c P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:46.296494 11232 ts_tablet_manager.cc:1367] T 738b8eadbc164d43bae145740a520d7c P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:46.296860 11232 raft_consensus.cc:348] T 738b8eadbc164d43bae145740a520d7c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:46.296962 11232 raft_consensus.cc:374] T 738b8eadbc164d43bae145740a520d7c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:46.296998 11232 raft_consensus.cc:729] T 738b8eadbc164d43bae145740a520d7c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:46.297134 11232 consensus_queue.cc:260] T 738b8eadbc164d43bae145740a520d7c P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:46.297413 11235 tablet_bootstrap.cc:492] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:46.298378 11235 tablet_bootstrap.cc:654] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:46.298441 11232 ts_tablet_manager.cc:1398] T 738b8eadbc164d43bae145740a520d7c P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:46.298485 11231 tablet_bootstrap.cc:492] T 738b8eadbc164d43bae145740a520d7c P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:46.298606 11231 ts_tablet_manager.cc:1367] T 738b8eadbc164d43bae145740a520d7c P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:46.299178 11231 raft_consensus.cc:348] T 738b8eadbc164d43bae145740a520d7c P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:46.299353 11231 raft_consensus.cc:374] T 738b8eadbc164d43bae145740a520d7c P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:46.299428 11231 raft_consensus.cc:729] T 738b8eadbc164d43bae145740a520d7c P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:46.299511 11235 tablet_bootstrap.cc:492] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:46.299618 11235 ts_tablet_manager.cc:1367] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:46.299573 11231 consensus_queue.cc:260] T 738b8eadbc164d43bae145740a520d7c P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:46.300374 11235 raft_consensus.cc:348] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:46.300499 11235 raft_consensus.cc:374] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:46.300556 11235 raft_consensus.cc:729] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:46.300695 11235 consensus_queue.cc:260] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:46.300884 11231 ts_tablet_manager.cc:1398] T 738b8eadbc164d43bae145740a520d7c P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:46.301028 11235 ts_tablet_manager.cc:1398] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:46.329095 11181 raft_consensus.cc:482] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:46.329205 11181 raft_consensus.cc:504] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:46.329639 11181 leader_election.cc:288] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:46.330014 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "738b8eadbc164d43bae145740a520d7c" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:46.330017 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "738b8eadbc164d43bae145740a520d7c" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:46.330153 9205 raft_consensus.cc:2454] T 738b8eadbc164d43bae145740a520d7c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:46.330154 6660 raft_consensus.cc:2454] T 738b8eadbc164d43bae145740a520d7c P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 0.
I20240503 13:59:46.330474 6465 leader_election.cc:302] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:46.330651 11181 raft_consensus.cc:2781] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:46.330710 11181 raft_consensus.cc:482] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:46.330745 11181 raft_consensus.cc:3037] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:46.331737 11181 raft_consensus.cc:504] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:46.332077 11181 leader_election.cc:288] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:46.332530 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "738b8eadbc164d43bae145740a520d7c" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:46.332553 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "738b8eadbc164d43bae145740a520d7c" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:46.332657 6660 raft_consensus.cc:3037] T 738b8eadbc164d43bae145740a520d7c P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:46.332659 9205 raft_consensus.cc:3037] T 738b8eadbc164d43bae145740a520d7c P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:46.333426 6660 raft_consensus.cc:2454] T 738b8eadbc164d43bae145740a520d7c P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:46.333436 9205 raft_consensus.cc:2454] T 738b8eadbc164d43bae145740a520d7c P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 13:59:46.333693 6465 leader_election.cc:302] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:46.333830 11181 raft_consensus.cc:2781] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:46.333917 11181 raft_consensus.cc:686] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Becoming Leader. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Running, Role: LEADER
I20240503 13:59:46.334070 11181 consensus_queue.cc:237] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:46.336478 6383 catalog_manager.cc:5484] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:46.436383 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:46.441290 6383 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:46.443090 9190 tablet_service.cc:1467] Processing CreateTablet for tablet e2ae8c7089db42199a2e6071800d9efc (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=2d921adc27d94e8bae2f87cf4caf89d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:46.443194 8926 tablet_service.cc:1467] Processing CreateTablet for tablet e2ae8c7089db42199a2e6071800d9efc (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=2d921adc27d94e8bae2f87cf4caf89d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:46.443249 9192 tablet_service.cc:1467] Processing CreateTablet for tablet fc0f85384e434c588213afd4272785c8 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=2d921adc27d94e8bae2f87cf4caf89d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:46.443413 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2ae8c7089db42199a2e6071800d9efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:46.443450 8926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2ae8c7089db42199a2e6071800d9efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:46.443532 9192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc0f85384e434c588213afd4272785c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:46.443571 6507 tablet_service.cc:1467] Processing CreateTablet for tablet e2ae8c7089db42199a2e6071800d9efc (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=2d921adc27d94e8bae2f87cf4caf89d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:46.443819 6507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2ae8c7089db42199a2e6071800d9efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:46.445654 11232 tablet_bootstrap.cc:492] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:46.446631 11232 tablet_bootstrap.cc:654] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:46.446808 11238 tablet_bootstrap.cc:492] T e2ae8c7089db42199a2e6071800d9efc P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:46.448184 9321 tablet_service.cc:1467] Processing CreateTablet for tablet fc0f85384e434c588213afd4272785c8 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=2d921adc27d94e8bae2f87cf4caf89d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:46.448477 9321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc0f85384e434c588213afd4272785c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:46.449299 9322 tablet_service.cc:1467] Processing CreateTablet for tablet 4a1398d983c54a7498257f766052e8e9 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=2d921adc27d94e8bae2f87cf4caf89d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:46.449510 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a1398d983c54a7498257f766052e8e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:46.451541 11240 tablet_bootstrap.cc:492] T fc0f85384e434c588213afd4272785c8 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:46.451892 6639 tablet_service.cc:1467] Processing CreateTablet for tablet 4a1398d983c54a7498257f766052e8e9 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=2d921adc27d94e8bae2f87cf4caf89d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:46.452183 6639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a1398d983c54a7498257f766052e8e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:46.452402 8926 tablet_service.cc:1467] Processing CreateTablet for tablet fc0f85384e434c588213afd4272785c8 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=2d921adc27d94e8bae2f87cf4caf89d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:46.452664 8926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc0f85384e434c588213afd4272785c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:46.452402 8928 tablet_service.cc:1467] Processing CreateTablet for tablet 4a1398d983c54a7498257f766052e8e9 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=2d921adc27d94e8bae2f87cf4caf89d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:46.452837 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a1398d983c54a7498257f766052e8e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:46.454305 11231 tablet_bootstrap.cc:492] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:46.455914 11235 tablet_bootstrap.cc:492] T e2ae8c7089db42199a2e6071800d9efc P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:46.456444 11231 tablet_bootstrap.cc:654] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:46.456984 9205 raft_consensus.cc:1261] T 738b8eadbc164d43bae145740a520d7c P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:46.457258 11235 tablet_bootstrap.cc:654] T e2ae8c7089db42199a2e6071800d9efc P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:46.457630 11181 consensus_queue.cc:1035] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:46.458453 11232 tablet_bootstrap.cc:492] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:46.458540 11240 tablet_bootstrap.cc:654] T fc0f85384e434c588213afd4272785c8 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:46.458536 11232 ts_tablet_manager.cc:1367] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 13:59:46.459019 11232 raft_consensus.cc:348] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:46.459112 11232 raft_consensus.cc:374] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:46.459141 11232 raft_consensus.cc:729] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:46.459254 11232 consensus_queue.cc:260] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:46.459630 11232 ts_tablet_manager.cc:1398] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:46.459777 11232 tablet_bootstrap.cc:492] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:46.460039 6660 raft_consensus.cc:1261] T 738b8eadbc164d43bae145740a520d7c P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:46.460579 11240 tablet_bootstrap.cc:492] T fc0f85384e434c588213afd4272785c8 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:46.460651 11232 tablet_bootstrap.cc:654] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:46.460592 11181 consensus_queue.cc:1035] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:46.460700 11240 ts_tablet_manager.cc:1367] T fc0f85384e434c588213afd4272785c8 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:59:46.460382 6383 catalog_manager.cc:5576] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091) does not have the latest schema for tablet 738b8eadbc164d43bae145740a520d7c (table rebalancer_dynamic_table [id=1f620bc625904450aaf239a47a874516]). Expected version 1 got 0
I20240503 13:59:46.461161 11240 raft_consensus.cc:348] T fc0f85384e434c588213afd4272785c8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:46.461249 11240 raft_consensus.cc:374] T fc0f85384e434c588213afd4272785c8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:46.461278 11240 raft_consensus.cc:729] T fc0f85384e434c588213afd4272785c8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:46.461380 11240 consensus_queue.cc:260] T fc0f85384e434c588213afd4272785c8 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:46.461944 11232 tablet_bootstrap.cc:492] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:46.462006 11232 ts_tablet_manager.cc:1367] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:46.462445 11232 raft_consensus.cc:348] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:46.462558 11232 raft_consensus.cc:374] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:46.462594 11232 raft_consensus.cc:729] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:46.462692 11232 consensus_queue.cc:260] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:46.462765 11240 ts_tablet_manager.cc:1398] T fc0f85384e434c588213afd4272785c8 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:46.462999 11240 tablet_bootstrap.cc:492] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:46.463024 11232 ts_tablet_manager.cc:1398] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:46.464157 11240 tablet_bootstrap.cc:654] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:46.464818 6380 catalog_manager.cc:5576] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119) does not have the latest schema for tablet 738b8eadbc164d43bae145740a520d7c (table rebalancer_dynamic_table [id=1f620bc625904450aaf239a47a874516]). Expected version 1 got 0
I20240503 13:59:46.466126 11235 tablet_bootstrap.cc:492] T e2ae8c7089db42199a2e6071800d9efc P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:46.466249 11235 ts_tablet_manager.cc:1367] T e2ae8c7089db42199a2e6071800d9efc P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 13:59:46.466718 11235 raft_consensus.cc:348] T e2ae8c7089db42199a2e6071800d9efc P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:46.466830 11235 raft_consensus.cc:374] T e2ae8c7089db42199a2e6071800d9efc P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:46.466897 11235 raft_consensus.cc:729] T e2ae8c7089db42199a2e6071800d9efc P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:46.467049 11235 consensus_queue.cc:260] T e2ae8c7089db42199a2e6071800d9efc P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:46.467504 11235 ts_tablet_manager.cc:1398] T e2ae8c7089db42199a2e6071800d9efc P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:46.468029 11215 tablet.cc:1722] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:46.468286 11215 tablet.cc:1620] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da: MemRowSet was empty: no flush needed.
I20240503 13:59:46.468639 11167 consensus_queue.cc:1035] T abe859639139483aa1e57ee6c9722b87 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:46.470875 11240 tablet_bootstrap.cc:492] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:46.470953 11240 ts_tablet_manager.cc:1367] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:59:46.471562 11240 raft_consensus.cc:348] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:46.471917 11240 raft_consensus.cc:374] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:46.471971 11240 raft_consensus.cc:729] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:46.471988 11238 tablet_bootstrap.cc:654] T e2ae8c7089db42199a2e6071800d9efc P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:46.472092 11240 consensus_queue.cc:260] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:46.472483 11240 ts_tablet_manager.cc:1398] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:46.475106 11238 tablet_bootstrap.cc:492] T e2ae8c7089db42199a2e6071800d9efc P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:46.475195 11238 ts_tablet_manager.cc:1367] T e2ae8c7089db42199a2e6071800d9efc P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.002s
I20240503 13:59:46.476012 11172 consensus_queue.cc:1035] T abe859639139483aa1e57ee6c9722b87 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:46.476686 11238 raft_consensus.cc:348] T e2ae8c7089db42199a2e6071800d9efc P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:46.476828 11238 raft_consensus.cc:374] T e2ae8c7089db42199a2e6071800d9efc P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:46.476917 11238 raft_consensus.cc:729] T e2ae8c7089db42199a2e6071800d9efc P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:46.477057 11238 consensus_queue.cc:260] T e2ae8c7089db42199a2e6071800d9efc P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:46.477380 11238 ts_tablet_manager.cc:1398] T e2ae8c7089db42199a2e6071800d9efc P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:46.477571 11238 tablet_bootstrap.cc:492] T 4a1398d983c54a7498257f766052e8e9 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:46.478489 11238 tablet_bootstrap.cc:654] T 4a1398d983c54a7498257f766052e8e9 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:46.480147 6385 catalog_manager.cc:5576] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119) does not have the latest schema for tablet 738b8eadbc164d43bae145740a520d7c (table rebalancer_dynamic_table [id=1f620bc625904450aaf239a47a874516]). Expected version 1 got 0
I20240503 13:59:46.485069 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=1f620bc625904450aaf239a47a874516] alter complete (version 1)
I20240503 13:59:46.486986 11253 tablet.cc:1722] T 738b8eadbc164d43bae145740a520d7c P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:46.487329 11253 tablet.cc:1620] T 738b8eadbc164d43bae145740a520d7c P 2145d15ab2d74c0287839266f1411a43: MemRowSet was empty: no flush needed.
I20240503 13:59:46.488274 11231 tablet_bootstrap.cc:492] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:46.488373 11231 ts_tablet_manager.cc:1367] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.000s
I20240503 13:59:46.489159 11231 raft_consensus.cc:348] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:46.489341 11231 raft_consensus.cc:374] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:46.489426 11231 raft_consensus.cc:729] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:46.489768 11231 consensus_queue.cc:260] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:46.490640 11231 ts_tablet_manager.cc:1398] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:46.494818 11238 tablet_bootstrap.cc:492] T 4a1398d983c54a7498257f766052e8e9 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:46.494927 11238 ts_tablet_manager.cc:1367] T 4a1398d983c54a7498257f766052e8e9 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.002s
I20240503 13:59:46.495396 11238 raft_consensus.cc:348] T 4a1398d983c54a7498257f766052e8e9 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:46.495594 11238 raft_consensus.cc:374] T 4a1398d983c54a7498257f766052e8e9 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:46.495663 11238 raft_consensus.cc:729] T 4a1398d983c54a7498257f766052e8e9 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:46.495842 11238 consensus_queue.cc:260] T 4a1398d983c54a7498257f766052e8e9 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:46.495872 6380 catalog_manager.cc:5576] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091) does not have the latest schema for tablet 738b8eadbc164d43bae145740a520d7c (table rebalancer_dynamic_table [id=1f620bc625904450aaf239a47a874516]). Expected version 1 got 0
I20240503 13:59:46.496239 11238 ts_tablet_manager.cc:1398] T 4a1398d983c54a7498257f766052e8e9 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:46.496611 11238 tablet_bootstrap.cc:492] T fc0f85384e434c588213afd4272785c8 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:46.497192 11256 tablet.cc:1722] T 738b8eadbc164d43bae145740a520d7c P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:46.497494 11256 tablet.cc:1620] T 738b8eadbc164d43bae145740a520d7c P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:46.497897 11238 tablet_bootstrap.cc:654] T fc0f85384e434c588213afd4272785c8 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:46.499007 11238 tablet_bootstrap.cc:492] T fc0f85384e434c588213afd4272785c8 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:46.499081 11238 ts_tablet_manager.cc:1367] T fc0f85384e434c588213afd4272785c8 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:46.499508 11238 raft_consensus.cc:348] T fc0f85384e434c588213afd4272785c8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:46.499619 11238 raft_consensus.cc:374] T fc0f85384e434c588213afd4272785c8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:46.499658 11238 raft_consensus.cc:729] T fc0f85384e434c588213afd4272785c8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:46.499764 11238 consensus_queue.cc:260] T fc0f85384e434c588213afd4272785c8 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:46.500036 11238 ts_tablet_manager.cc:1398] T fc0f85384e434c588213afd4272785c8 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:46.587494 6385 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:46.592463 11215 tablet.cc:1722] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:46.592808 11215 tablet.cc:1620] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da: MemRowSet was empty: no flush needed.
I20240503 13:59:46.592849 11253 tablet.cc:1722] T 738b8eadbc164d43bae145740a520d7c P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:46.593123 11253 tablet.cc:1620] T 738b8eadbc164d43bae145740a520d7c P 2145d15ab2d74c0287839266f1411a43: MemRowSet was empty: no flush needed.
I20240503 13:59:46.593573 11256 tablet.cc:1722] T 738b8eadbc164d43bae145740a520d7c P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:46.593856 11256 tablet.cc:1620] T 738b8eadbc164d43bae145740a520d7c P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:46.599702 6380 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=1f620bc625904450aaf239a47a874516] alter complete (version 2)
I20240503 13:59:46.600301 10942 raft_consensus.cc:482] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:46.600385 10942 raft_consensus.cc:504] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:46.600677 10942 leader_election.cc:288] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:46.601284 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ae8c7089db42199a2e6071800d9efc" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:46.601425 8948 raft_consensus.cc:2454] T e2ae8c7089db42199a2e6071800d9efc P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:46.602226 9147 leader_election.cc:302] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:46.602464 10942 raft_consensus.cc:2781] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:46.602525 10942 raft_consensus.cc:482] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:46.602564 10942 raft_consensus.cc:3037] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:46.603133 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ae8c7089db42199a2e6071800d9efc" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:46.603250 6527 raft_consensus.cc:2454] T e2ae8c7089db42199a2e6071800d9efc P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:46.603654 10942 raft_consensus.cc:504] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:46.604010 10942 leader_election.cc:288] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:46.605089 8946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ae8c7089db42199a2e6071800d9efc" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:46.605211 8946 raft_consensus.cc:3037] T e2ae8c7089db42199a2e6071800d9efc P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:46.605832 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ae8c7089db42199a2e6071800d9efc" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:46.605943 6527 raft_consensus.cc:3037] T e2ae8c7089db42199a2e6071800d9efc P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:46.606334 8946 raft_consensus.cc:2454] T e2ae8c7089db42199a2e6071800d9efc P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:46.606627 9147 leader_election.cc:302] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:46.606844 10942 raft_consensus.cc:2781] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:46.606999 10942 raft_consensus.cc:686] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 13:59:46.607163 10942 consensus_queue.cc:237] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:46.609383 6383 catalog_manager.cc:5484] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:46.606951 6527 raft_consensus.cc:2454] T e2ae8c7089db42199a2e6071800d9efc P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:46.706655 6383 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:46.711719 11215 tablet.cc:1722] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:46.712147 11256 tablet.cc:1722] T 738b8eadbc164d43bae145740a520d7c P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:46.712203 11253 tablet.cc:1722] T 738b8eadbc164d43bae145740a520d7c P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:46.715763 10942 raft_consensus.cc:482] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:46.715847 10942 raft_consensus.cc:504] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:46.716451 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc0f85384e434c588213afd4272785c8" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:46.716614 9343 raft_consensus.cc:2454] T fc0f85384e434c588213afd4272785c8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:46.717046 9148 leader_election.cc:302] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:46.717096 8946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc0f85384e434c588213afd4272785c8" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:46.717229 6383 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=1f620bc625904450aaf239a47a874516] alter complete (version 3)
I20240503 13:59:46.717242 8946 raft_consensus.cc:2454] T fc0f85384e434c588213afd4272785c8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:46.717289 11071 raft_consensus.cc:2781] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:46.717350 11071 raft_consensus.cc:482] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:46.717442 11071 raft_consensus.cc:3037] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:46.718477 11071 raft_consensus.cc:504] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:46.718811 11071 leader_election.cc:288] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:46.719187 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc0f85384e434c588213afd4272785c8" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:46.719306 9343 raft_consensus.cc:3037] T fc0f85384e434c588213afd4272785c8 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:46.719296 8946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc0f85384e434c588213afd4272785c8" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:46.719410 8946 raft_consensus.cc:3037] T fc0f85384e434c588213afd4272785c8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:46.720094 9343 raft_consensus.cc:2454] T fc0f85384e434c588213afd4272785c8 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:46.720283 8946 raft_consensus.cc:2454] T fc0f85384e434c588213afd4272785c8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:46.720410 9148 leader_election.cc:302] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:46.720577 11071 raft_consensus.cc:2781] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:46.720664 11071 raft_consensus.cc:686] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 13:59:46.720782 11071 consensus_queue.cc:237] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:46.721216 10942 leader_election.cc:288] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:46.722784 6385 catalog_manager.cc:5484] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:46.757193 11258 raft_consensus.cc:482] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:46.757305 11258 raft_consensus.cc:504] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:46.757663 11258 leader_election.cc:288] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:46.757951 8946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1398d983c54a7498257f766052e8e9" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:46.757987 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1398d983c54a7498257f766052e8e9" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:46.758086 8946 raft_consensus.cc:2454] T 4a1398d983c54a7498257f766052e8e9 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:46.758100 6660 raft_consensus.cc:2454] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:46.758343 9279 leader_election.cc:302] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 2145d15ab2d74c0287839266f1411a43; no voters:
I20240503 13:59:46.758538 11258 raft_consensus.cc:2781] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:46.758597 11258 raft_consensus.cc:482] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:46.758623 11258 raft_consensus.cc:3037] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:46.759469 11258 raft_consensus.cc:504] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:46.759776 11258 leader_election.cc:288] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:46.760056 8946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1398d983c54a7498257f766052e8e9" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:46.760052 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1398d983c54a7498257f766052e8e9" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:46.760159 8946 raft_consensus.cc:3037] T 4a1398d983c54a7498257f766052e8e9 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:46.760156 6660 raft_consensus.cc:3037] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:46.760885 6660 raft_consensus.cc:2454] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:46.760939 8946 raft_consensus.cc:2454] T 4a1398d983c54a7498257f766052e8e9 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:46.761157 9279 leader_election.cc:302] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 2145d15ab2d74c0287839266f1411a43; no voters:
I20240503 13:59:46.761327 11258 raft_consensus.cc:2781] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:46.761425 11258 raft_consensus.cc:686] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:46.761576 11258 consensus_queue.cc:237] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:46.763940 6385 catalog_manager.cc:5484] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:46.819058 6385 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:46.819221 6385 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:46.821528 6385 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet 738b8eadbc164d43bae145740a520d7c
I20240503 13:59:46.822134 6639 tablet_service.cc:1514] Processing DeleteTablet for tablet 738b8eadbc164d43bae145740a520d7c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:46 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:46.822446 11263 tablet_replica.cc:330] T 738b8eadbc164d43bae145740a520d7c P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:46.822449 6507 tablet_service.cc:1514] Processing DeleteTablet for tablet 738b8eadbc164d43bae145740a520d7c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:46 UTC) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:46.822476 9192 tablet_service.cc:1514] Processing DeleteTablet for tablet 738b8eadbc164d43bae145740a520d7c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:46 UTC) from {username='slave'} at 127.0.0.1:57960
I20240503 13:59:46.822775 11265 tablet_replica.cc:330] T 738b8eadbc164d43bae145740a520d7c P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 13:59:46.822794 11263 raft_consensus.cc:2229] T 738b8eadbc164d43bae145740a520d7c P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:46.822942 11265 raft_consensus.cc:2229] T 738b8eadbc164d43bae145740a520d7c P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:46.822957 11263 raft_consensus.cc:2258] T 738b8eadbc164d43bae145740a520d7c P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:46.823064 11265 raft_consensus.cc:2258] T 738b8eadbc164d43bae145740a520d7c P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:46.823669 11263 ts_tablet_manager.cc:1875] T 738b8eadbc164d43bae145740a520d7c P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:46.823663 11265 ts_tablet_manager.cc:1875] T 738b8eadbc164d43bae145740a520d7c P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:46.825531 11263 ts_tablet_manager.cc:1888] T 738b8eadbc164d43bae145740a520d7c P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:46.825608 11263 log.cc:1196] T 738b8eadbc164d43bae145740a520d7c P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/738b8eadbc164d43bae145740a520d7c
I20240503 13:59:46.825901 11263 ts_tablet_manager.cc:1909] T 738b8eadbc164d43bae145740a520d7c P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:46.825914 11265 ts_tablet_manager.cc:1888] T 738b8eadbc164d43bae145740a520d7c P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:46.825979 11265 log.cc:1196] T 738b8eadbc164d43bae145740a520d7c P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/738b8eadbc164d43bae145740a520d7c
I20240503 13:59:46.826360 11265 ts_tablet_manager.cc:1909] T 738b8eadbc164d43bae145740a520d7c P ab1d21935d6049888eb0ceccc3c9d735: Deleting consensus metadata
I20240503 13:59:46.826615 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet 738b8eadbc164d43bae145740a520d7c (table rebalancer_dynamic_table_new_name [id=1f620bc625904450aaf239a47a874516]) successfully deleted
I20240503 13:59:46.826992 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet 738b8eadbc164d43bae145740a520d7c (table rebalancer_dynamic_table_new_name [id=1f620bc625904450aaf239a47a874516]) successfully deleted
I20240503 13:59:46.835448 11264 tablet_replica.cc:330] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:46.835616 11264 raft_consensus.cc:2229] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:46.835840 11264 raft_consensus.cc:2258] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:46.837095 11264 ts_tablet_manager.cc:1875] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:46.839183 11264 ts_tablet_manager.cc:1888] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:46.839260 11264 log.cc:1196] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/738b8eadbc164d43bae145740a520d7c
I20240503 13:59:46.839555 11264 ts_tablet_manager.cc:1909] T 738b8eadbc164d43bae145740a520d7c P 5f46e19644fd4f4b825c86ba6af768da: Deleting consensus metadata
I20240503 13:59:46.840201 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 738b8eadbc164d43bae145740a520d7c (table rebalancer_dynamic_table_new_name [id=1f620bc625904450aaf239a47a874516]) successfully deleted
I20240503 13:59:46.923305 6385 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:46.928565 6771 tablet_service.cc:1467] Processing CreateTablet for tablet acaabafbf03345b8849f0cbd0a7d4347 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=47f3ce1454cc430db458ea8c511ea0f2]), partition=
I20240503 13:59:46.928750 9060 tablet_service.cc:1467] Processing CreateTablet for tablet acaabafbf03345b8849f0cbd0a7d4347 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=47f3ce1454cc430db458ea8c511ea0f2]), partition=
I20240503 13:59:46.928848 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acaabafbf03345b8849f0cbd0a7d4347. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:46.928974 9060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acaabafbf03345b8849f0cbd0a7d4347. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:46.930691 11266 tablet_bootstrap.cc:492] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:46.931409 11267 tablet_bootstrap.cc:492] T acaabafbf03345b8849f0cbd0a7d4347 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:46.931425 9321 tablet_service.cc:1467] Processing CreateTablet for tablet acaabafbf03345b8849f0cbd0a7d4347 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=47f3ce1454cc430db458ea8c511ea0f2]), partition=
I20240503 13:59:46.931653 9321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acaabafbf03345b8849f0cbd0a7d4347. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:46.931900 11266 tablet_bootstrap.cc:654] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:46.932251 11267 tablet_bootstrap.cc:654] T acaabafbf03345b8849f0cbd0a7d4347 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:46.933727 11240 tablet_bootstrap.cc:492] T acaabafbf03345b8849f0cbd0a7d4347 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:46.934615 11266 tablet_bootstrap.cc:492] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:46.934697 11266 ts_tablet_manager.cc:1367] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:46.935067 11240 tablet_bootstrap.cc:654] T acaabafbf03345b8849f0cbd0a7d4347 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:46.935020 11266 raft_consensus.cc:348] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:46.935101 11266 raft_consensus.cc:374] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:46.935120 11266 raft_consensus.cc:729] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:46.935216 11266 consensus_queue.cc:260] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:46.935480 11266 ts_tablet_manager.cc:1398] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:46.936434 11267 tablet_bootstrap.cc:492] T acaabafbf03345b8849f0cbd0a7d4347 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:46.936537 11267 ts_tablet_manager.cc:1367] T acaabafbf03345b8849f0cbd0a7d4347 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:46.937263 11267 raft_consensus.cc:348] T acaabafbf03345b8849f0cbd0a7d4347 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:46.937356 11267 raft_consensus.cc:374] T acaabafbf03345b8849f0cbd0a7d4347 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:46.937383 11267 raft_consensus.cc:729] T acaabafbf03345b8849f0cbd0a7d4347 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:46.937497 11267 consensus_queue.cc:260] T acaabafbf03345b8849f0cbd0a7d4347 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:46.937731 11267 ts_tablet_manager.cc:1398] T acaabafbf03345b8849f0cbd0a7d4347 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:46.941038 11240 tablet_bootstrap.cc:492] T acaabafbf03345b8849f0cbd0a7d4347 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:46.941115 11240 ts_tablet_manager.cc:1367] T acaabafbf03345b8849f0cbd0a7d4347 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240503 13:59:46.941481 11240 raft_consensus.cc:348] T acaabafbf03345b8849f0cbd0a7d4347 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:46.941566 11240 raft_consensus.cc:374] T acaabafbf03345b8849f0cbd0a7d4347 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:46.941593 11240 raft_consensus.cc:729] T acaabafbf03345b8849f0cbd0a7d4347 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:46.941701 11240 consensus_queue.cc:260] T acaabafbf03345b8849f0cbd0a7d4347 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:46.941969 11240 ts_tablet_manager.cc:1398] T acaabafbf03345b8849f0cbd0a7d4347 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:47.072680 10942 consensus_queue.cc:1035] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:47.075670 11260 consensus_queue.cc:1035] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:47.125761 11205 raft_consensus.cc:482] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:47.125895 11205 raft_consensus.cc:504] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:47.126399 11205 leader_election.cc:288] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:47.126662 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acaabafbf03345b8849f0cbd0a7d4347" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:47.126793 9081 raft_consensus.cc:2454] T acaabafbf03345b8849f0cbd0a7d4347 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:47.126844 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acaabafbf03345b8849f0cbd0a7d4347" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:47.126966 9343 raft_consensus.cc:2454] T acaabafbf03345b8849f0cbd0a7d4347 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:47.127116 6726 leader_election.cc:302] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:47.127322 11205 raft_consensus.cc:2781] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:47.127394 11205 raft_consensus.cc:482] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:47.127446 11205 raft_consensus.cc:3037] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:47.128293 11205 raft_consensus.cc:504] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:47.128568 11205 leader_election.cc:288] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:47.128913 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acaabafbf03345b8849f0cbd0a7d4347" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:47.128957 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acaabafbf03345b8849f0cbd0a7d4347" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:47.129040 9081 raft_consensus.cc:3037] T acaabafbf03345b8849f0cbd0a7d4347 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:47.129058 9343 raft_consensus.cc:3037] T acaabafbf03345b8849f0cbd0a7d4347 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:47.129842 9081 raft_consensus.cc:2454] T acaabafbf03345b8849f0cbd0a7d4347 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:47.129958 9343 raft_consensus.cc:2454] T acaabafbf03345b8849f0cbd0a7d4347 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:47.130156 6726 leader_election.cc:302] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:47.130594 11205 raft_consensus.cc:2781] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:47.130678 11205 raft_consensus.cc:686] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:47.130805 11205 consensus_queue.cc:237] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:47.133194 6383 catalog_manager.cc:5484] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:47.168851 10942 consensus_queue.cc:1035] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:47.172209 11260 consensus_queue.cc:1035] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:47.178511 11172 consensus_queue.cc:1035] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:47.183008 11172 consensus_queue.cc:1035] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:47.266531 6383 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:47.270470 9342 raft_consensus.cc:1261] T acaabafbf03345b8849f0cbd0a7d4347 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:47.271248 11205 consensus_queue.cc:1035] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:47.271701 9081 raft_consensus.cc:1261] T acaabafbf03345b8849f0cbd0a7d4347 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:47.272663 11205 consensus_queue.cc:1035] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:47.273070 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:47.277202 6380 catalog_manager.cc:5576] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931) does not have the latest schema for tablet acaabafbf03345b8849f0cbd0a7d4347 (table rebalancer_dynamic_table [id=47f3ce1454cc430db458ea8c511ea0f2]). Expected version 1 got 0
I20240503 13:59:47.278116 6379 catalog_manager.cc:5576] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481) does not have the latest schema for tablet acaabafbf03345b8849f0cbd0a7d4347 (table rebalancer_dynamic_table [id=47f3ce1454cc430db458ea8c511ea0f2]). Expected version 1 got 0
I20240503 13:59:47.280230 11288 tablet.cc:1722] T acaabafbf03345b8849f0cbd0a7d4347 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:47.280560 11288 tablet.cc:1620] T acaabafbf03345b8849f0cbd0a7d4347 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:47.280913 6379 catalog_manager.cc:5576] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931) does not have the latest schema for tablet acaabafbf03345b8849f0cbd0a7d4347 (table rebalancer_dynamic_table [id=47f3ce1454cc430db458ea8c511ea0f2]). Expected version 1 got 0
I20240503 13:59:47.282339 6385 catalog_manager.cc:5576] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365) does not have the latest schema for tablet acaabafbf03345b8849f0cbd0a7d4347 (table rebalancer_dynamic_table [id=47f3ce1454cc430db458ea8c511ea0f2]). Expected version 1 got 0
I20240503 13:59:47.284358 6901 tablet_service.cc:1467] Processing CreateTablet for tablet f403dc06195d449e8f454a9f4e795925 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=0d758a66160e4c21a4e6c8f5b3cb6179]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:47.284685 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f403dc06195d449e8f454a9f4e795925. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:47.284719 6507 tablet_service.cc:1467] Processing CreateTablet for tablet f403dc06195d449e8f454a9f4e795925 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=0d758a66160e4c21a4e6c8f5b3cb6179]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:47.285023 6507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f403dc06195d449e8f454a9f4e795925. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:47.285120 9192 tablet_service.cc:1467] Processing CreateTablet for tablet 075e3dc3a3de4a42a6a3301b821f1f3d (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=0d758a66160e4c21a4e6c8f5b3cb6179]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:47.285356 9192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 075e3dc3a3de4a42a6a3301b821f1f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:47.287567 9061 tablet_service.cc:1467] Processing CreateTablet for tablet af8bba02684c4555893def9c1e88ebf0 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=0d758a66160e4c21a4e6c8f5b3cb6179]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:47.287573 8926 tablet_service.cc:1467] Processing CreateTablet for tablet 075e3dc3a3de4a42a6a3301b821f1f3d (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=0d758a66160e4c21a4e6c8f5b3cb6179]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:47.287832 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af8bba02684c4555893def9c1e88ebf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:47.287842 8926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 075e3dc3a3de4a42a6a3301b821f1f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:47.288587 11289 tablet.cc:1722] T acaabafbf03345b8849f0cbd0a7d4347 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:47.288868 11289 tablet.cc:1620] T acaabafbf03345b8849f0cbd0a7d4347 P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:47.289220 11287 tablet.cc:1722] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:47.289477 11287 tablet.cc:1620] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64: MemRowSet was empty: no flush needed.
I20240503 13:59:47.289862 11267 tablet_bootstrap.cc:492] T af8bba02684c4555893def9c1e88ebf0 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:47.290196 9060 tablet_service.cc:1467] Processing CreateTablet for tablet 075e3dc3a3de4a42a6a3301b821f1f3d (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=0d758a66160e4c21a4e6c8f5b3cb6179]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:47.290426 9060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 075e3dc3a3de4a42a6a3301b821f1f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:47.290858 11267 tablet_bootstrap.cc:654] T af8bba02684c4555893def9c1e88ebf0 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:47.291558 6900 tablet_service.cc:1467] Processing CreateTablet for tablet af8bba02684c4555893def9c1e88ebf0 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=0d758a66160e4c21a4e6c8f5b3cb6179]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:47.291824 6900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af8bba02684c4555893def9c1e88ebf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:47.291757 6640 tablet_service.cc:1467] Processing CreateTablet for tablet af8bba02684c4555893def9c1e88ebf0 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=0d758a66160e4c21a4e6c8f5b3cb6179]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:47.292064 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af8bba02684c4555893def9c1e88ebf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:47.294312 6639 tablet_service.cc:1467] Processing CreateTablet for tablet f403dc06195d449e8f454a9f4e795925 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=0d758a66160e4c21a4e6c8f5b3cb6179]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:47.294553 6639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f403dc06195d449e8f454a9f4e795925. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:47.295506 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=47f3ce1454cc430db458ea8c511ea0f2] alter complete (version 1)
I20240503 13:59:47.297061 11295 tablet_bootstrap.cc:492] T 075e3dc3a3de4a42a6a3301b821f1f3d P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:47.297698 11291 tablet_bootstrap.cc:492] T f403dc06195d449e8f454a9f4e795925 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:47.299423 11294 tablet_bootstrap.cc:492] T af8bba02684c4555893def9c1e88ebf0 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:47.300452 11267 tablet_bootstrap.cc:492] T af8bba02684c4555893def9c1e88ebf0 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:47.300545 11267 ts_tablet_manager.cc:1367] T af8bba02684c4555893def9c1e88ebf0 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 13:59:47.300639 11294 tablet_bootstrap.cc:654] T af8bba02684c4555893def9c1e88ebf0 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:47.301123 11292 tablet_bootstrap.cc:492] T 075e3dc3a3de4a42a6a3301b821f1f3d P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:47.301079 11267 raft_consensus.cc:348] T af8bba02684c4555893def9c1e88ebf0 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:47.301193 11267 raft_consensus.cc:374] T af8bba02684c4555893def9c1e88ebf0 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:47.301227 11267 raft_consensus.cc:729] T af8bba02684c4555893def9c1e88ebf0 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:47.301347 11267 consensus_queue.cc:260] T af8bba02684c4555893def9c1e88ebf0 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:47.301436 11291 tablet_bootstrap.cc:654] T f403dc06195d449e8f454a9f4e795925 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:47.301647 11267 ts_tablet_manager.cc:1398] T af8bba02684c4555893def9c1e88ebf0 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:47.301815 11267 tablet_bootstrap.cc:492] T 075e3dc3a3de4a42a6a3301b821f1f3d P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:47.302896 11267 tablet_bootstrap.cc:654] T 075e3dc3a3de4a42a6a3301b821f1f3d P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:47.305590 11295 tablet_bootstrap.cc:654] T 075e3dc3a3de4a42a6a3301b821f1f3d P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:47.305788 11267 tablet_bootstrap.cc:492] T 075e3dc3a3de4a42a6a3301b821f1f3d P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:47.305851 11267 ts_tablet_manager.cc:1367] T 075e3dc3a3de4a42a6a3301b821f1f3d P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:47.306233 11267 raft_consensus.cc:348] T 075e3dc3a3de4a42a6a3301b821f1f3d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:47.306319 11267 raft_consensus.cc:374] T 075e3dc3a3de4a42a6a3301b821f1f3d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:47.306345 11267 raft_consensus.cc:729] T 075e3dc3a3de4a42a6a3301b821f1f3d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:47.306524 11295 tablet_bootstrap.cc:492] T 075e3dc3a3de4a42a6a3301b821f1f3d P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:47.306492 11267 consensus_queue.cc:260] T 075e3dc3a3de4a42a6a3301b821f1f3d P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:47.306586 11295 ts_tablet_manager.cc:1367] T 075e3dc3a3de4a42a6a3301b821f1f3d P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 13:59:47.306728 11267 ts_tablet_manager.cc:1398] T 075e3dc3a3de4a42a6a3301b821f1f3d P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:47.307515 11292 tablet_bootstrap.cc:654] T 075e3dc3a3de4a42a6a3301b821f1f3d P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:47.307662 11295 raft_consensus.cc:348] T 075e3dc3a3de4a42a6a3301b821f1f3d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:47.307840 11295 raft_consensus.cc:374] T 075e3dc3a3de4a42a6a3301b821f1f3d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:47.307924 11295 raft_consensus.cc:729] T 075e3dc3a3de4a42a6a3301b821f1f3d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:47.308120 11295 consensus_queue.cc:260] T 075e3dc3a3de4a42a6a3301b821f1f3d P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:47.308518 11295 ts_tablet_manager.cc:1398] T 075e3dc3a3de4a42a6a3301b821f1f3d P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:47.309255 11290 tablet_bootstrap.cc:492] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:47.310772 11290 tablet_bootstrap.cc:654] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:47.312351 11292 tablet_bootstrap.cc:492] T 075e3dc3a3de4a42a6a3301b821f1f3d P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:47.312676 11292 ts_tablet_manager.cc:1367] T 075e3dc3a3de4a42a6a3301b821f1f3d P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 13:59:47.313217 11292 raft_consensus.cc:348] T 075e3dc3a3de4a42a6a3301b821f1f3d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:47.314637 11292 raft_consensus.cc:374] T 075e3dc3a3de4a42a6a3301b821f1f3d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:47.314738 11292 raft_consensus.cc:729] T 075e3dc3a3de4a42a6a3301b821f1f3d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:47.314914 11292 consensus_queue.cc:260] T 075e3dc3a3de4a42a6a3301b821f1f3d P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:47.315249 11292 ts_tablet_manager.cc:1398] T 075e3dc3a3de4a42a6a3301b821f1f3d P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:47.316203 11294 tablet_bootstrap.cc:492] T af8bba02684c4555893def9c1e88ebf0 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:47.316262 11290 tablet_bootstrap.cc:492] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:47.316300 11294 ts_tablet_manager.cc:1367] T af8bba02684c4555893def9c1e88ebf0 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240503 13:59:47.316342 11290 ts_tablet_manager.cc:1367] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 13:59:47.316712 11290 raft_consensus.cc:348] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:47.316807 11290 raft_consensus.cc:374] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:47.316836 11290 raft_consensus.cc:729] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:47.316768 11294 raft_consensus.cc:348] T af8bba02684c4555893def9c1e88ebf0 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:47.316872 11294 raft_consensus.cc:374] T af8bba02684c4555893def9c1e88ebf0 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:47.316915 11294 raft_consensus.cc:729] T af8bba02684c4555893def9c1e88ebf0 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:47.316982 11290 consensus_queue.cc:260] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:47.317047 11294 consensus_queue.cc:260] T af8bba02684c4555893def9c1e88ebf0 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:47.317229 11290 ts_tablet_manager.cc:1398] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:47.317350 11294 ts_tablet_manager.cc:1398] T af8bba02684c4555893def9c1e88ebf0 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:47.318321 11290 tablet_bootstrap.cc:492] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:47.318915 11294 tablet_bootstrap.cc:492] T f403dc06195d449e8f454a9f4e795925 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:47.319360 11290 tablet_bootstrap.cc:654] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:47.319890 11294 tablet_bootstrap.cc:654] T f403dc06195d449e8f454a9f4e795925 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:47.322940 11291 tablet_bootstrap.cc:492] T f403dc06195d449e8f454a9f4e795925 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:47.323048 11291 ts_tablet_manager.cc:1367] T f403dc06195d449e8f454a9f4e795925 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240503 13:59:47.323568 11291 raft_consensus.cc:348] T f403dc06195d449e8f454a9f4e795925 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:47.323686 11291 raft_consensus.cc:374] T f403dc06195d449e8f454a9f4e795925 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:47.323719 11291 raft_consensus.cc:729] T f403dc06195d449e8f454a9f4e795925 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:47.323832 11291 consensus_queue.cc:260] T f403dc06195d449e8f454a9f4e795925 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:47.324115 11291 ts_tablet_manager.cc:1398] T f403dc06195d449e8f454a9f4e795925 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:47.324172 11290 tablet_bootstrap.cc:492] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:47.324242 11290 ts_tablet_manager.cc:1367] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240503 13:59:47.324807 11290 raft_consensus.cc:348] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:47.325515 11290 raft_consensus.cc:374] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:47.325603 11290 raft_consensus.cc:729] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:47.325611 11294 tablet_bootstrap.cc:492] T f403dc06195d449e8f454a9f4e795925 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:47.325690 11294 ts_tablet_manager.cc:1367] T f403dc06195d449e8f454a9f4e795925 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:59:47.325789 11290 consensus_queue.cc:260] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:47.327091 11290 ts_tablet_manager.cc:1398] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:59:47.326547 11294 raft_consensus.cc:348] T f403dc06195d449e8f454a9f4e795925 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:47.327432 11294 raft_consensus.cc:374] T f403dc06195d449e8f454a9f4e795925 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:47.327561 11294 raft_consensus.cc:729] T f403dc06195d449e8f454a9f4e795925 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:47.328928 11294 consensus_queue.cc:260] T f403dc06195d449e8f454a9f4e795925 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:47.329361 11294 ts_tablet_manager.cc:1398] T f403dc06195d449e8f454a9f4e795925 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:47.364681 11206 raft_consensus.cc:482] T 075e3dc3a3de4a42a6a3301b821f1f3d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:47.364815 11206 raft_consensus.cc:504] T 075e3dc3a3de4a42a6a3301b821f1f3d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:47.365234 11206 leader_election.cc:288] T 075e3dc3a3de4a42a6a3301b821f1f3d P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:47.365531 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "075e3dc3a3de4a42a6a3301b821f1f3d" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:47.365664 9205 raft_consensus.cc:2454] T 075e3dc3a3de4a42a6a3301b821f1f3d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:47.365635 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "075e3dc3a3de4a42a6a3301b821f1f3d" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:47.365749 9081 raft_consensus.cc:2454] T 075e3dc3a3de4a42a6a3301b821f1f3d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:47.365944 8883 leader_election.cc:302] T 075e3dc3a3de4a42a6a3301b821f1f3d P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:47.366159 11206 raft_consensus.cc:2781] T 075e3dc3a3de4a42a6a3301b821f1f3d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:47.366233 11206 raft_consensus.cc:482] T 075e3dc3a3de4a42a6a3301b821f1f3d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:47.366281 11206 raft_consensus.cc:3037] T 075e3dc3a3de4a42a6a3301b821f1f3d P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:47.367141 11206 raft_consensus.cc:504] T 075e3dc3a3de4a42a6a3301b821f1f3d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:47.367472 11206 leader_election.cc:288] T 075e3dc3a3de4a42a6a3301b821f1f3d P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:47.367767 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "075e3dc3a3de4a42a6a3301b821f1f3d" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:47.367830 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "075e3dc3a3de4a42a6a3301b821f1f3d" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:47.367884 9205 raft_consensus.cc:3037] T 075e3dc3a3de4a42a6a3301b821f1f3d P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:47.367938 9080 raft_consensus.cc:3037] T 075e3dc3a3de4a42a6a3301b821f1f3d P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:47.368911 9080 raft_consensus.cc:2454] T 075e3dc3a3de4a42a6a3301b821f1f3d P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:47.368911 9205 raft_consensus.cc:2454] T 075e3dc3a3de4a42a6a3301b821f1f3d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:47.369194 8883 leader_election.cc:302] T 075e3dc3a3de4a42a6a3301b821f1f3d P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:47.369360 11206 raft_consensus.cc:2781] T 075e3dc3a3de4a42a6a3301b821f1f3d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:47.369578 11206 raft_consensus.cc:686] T 075e3dc3a3de4a42a6a3301b821f1f3d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 13:59:47.369719 11206 consensus_queue.cc:237] T 075e3dc3a3de4a42a6a3301b821f1f3d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:47.371727 6379 catalog_manager.cc:5484] T 075e3dc3a3de4a42a6a3301b821f1f3d P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:47.399380 6379 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:47.404611 11287 tablet.cc:1722] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:47.404958 11287 tablet.cc:1620] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64: MemRowSet was empty: no flush needed.
I20240503 13:59:47.405732 11288 tablet.cc:1722] T acaabafbf03345b8849f0cbd0a7d4347 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:47.406033 11288 tablet.cc:1620] T acaabafbf03345b8849f0cbd0a7d4347 P 0d77752181e944698e809e325e5937e6: MemRowSet was empty: no flush needed.
I20240503 13:59:47.408161 11289 tablet.cc:1722] T acaabafbf03345b8849f0cbd0a7d4347 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:47.408433 11289 tablet.cc:1620] T acaabafbf03345b8849f0cbd0a7d4347 P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:47.412876 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=47f3ce1454cc430db458ea8c511ea0f2] alter complete (version 2)
I20240503 13:59:47.511952 11271 raft_consensus.cc:482] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:47.512079 11271 raft_consensus.cc:504] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:47.512506 11271 leader_election.cc:288] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:47.512755 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8bba02684c4555893def9c1e88ebf0" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:47.512813 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8bba02684c4555893def9c1e88ebf0" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:47.512890 9080 raft_consensus.cc:2454] T af8bba02684c4555893def9c1e88ebf0 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:47.512935 6660 raft_consensus.cc:2454] T af8bba02684c4555893def9c1e88ebf0 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:47.513217 6857 leader_election.cc:302] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:47.513414 11271 raft_consensus.cc:2781] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:47.513461 11271 raft_consensus.cc:482] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:47.513528 11271 raft_consensus.cc:3037] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:47.514636 11271 raft_consensus.cc:504] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:47.514948 11271 leader_election.cc:288] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:47.515249 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8bba02684c4555893def9c1e88ebf0" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:47.515270 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8bba02684c4555893def9c1e88ebf0" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:47.515393 9080 raft_consensus.cc:3037] T af8bba02684c4555893def9c1e88ebf0 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:47.515400 6660 raft_consensus.cc:3037] T af8bba02684c4555893def9c1e88ebf0 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:47.516121 9080 raft_consensus.cc:2454] T af8bba02684c4555893def9c1e88ebf0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:47.516197 6660 raft_consensus.cc:2454] T af8bba02684c4555893def9c1e88ebf0 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:47.516470 6857 leader_election.cc:302] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:47.516691 11271 raft_consensus.cc:2781] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:47.516860 11271 raft_consensus.cc:686] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:47.516817 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:47.516993 11271 consensus_queue.cc:237] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:47.519433 6379 catalog_manager.cc:5484] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:47.524214 11287 tablet.cc:1722] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:47.524688 11289 tablet.cc:1722] T acaabafbf03345b8849f0cbd0a7d4347 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:47.525009 11288 tablet.cc:1722] T acaabafbf03345b8849f0cbd0a7d4347 P 0d77752181e944698e809e325e5937e6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:47.529444 6379 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=47f3ce1454cc430db458ea8c511ea0f2] alter complete (version 3)
I20240503 13:59:47.631311 6380 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:47.631467 6380 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:47.633852 6380 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet acaabafbf03345b8849f0cbd0a7d4347
I20240503 13:59:47.634481 9321 tablet_service.cc:1514] Processing DeleteTablet for tablet acaabafbf03345b8849f0cbd0a7d4347 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:47 UTC) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:47.634737 9060 tablet_service.cc:1514] Processing DeleteTablet for tablet acaabafbf03345b8849f0cbd0a7d4347 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:47 UTC) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:47.635071 11309 tablet_replica.cc:330] T acaabafbf03345b8849f0cbd0a7d4347 P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:47.635200 11308 tablet_replica.cc:330] T acaabafbf03345b8849f0cbd0a7d4347 P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:47.635246 11309 raft_consensus.cc:2229] T acaabafbf03345b8849f0cbd0a7d4347 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:47.635380 11308 raft_consensus.cc:2229] T acaabafbf03345b8849f0cbd0a7d4347 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:47.635457 11309 raft_consensus.cc:2258] T acaabafbf03345b8849f0cbd0a7d4347 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:47.635507 11308 raft_consensus.cc:2258] T acaabafbf03345b8849f0cbd0a7d4347 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:47.636143 11309 ts_tablet_manager.cc:1875] T acaabafbf03345b8849f0cbd0a7d4347 P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:47.636591 11308 ts_tablet_manager.cc:1875] T acaabafbf03345b8849f0cbd0a7d4347 P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:47.636971 6771 tablet_service.cc:1514] Processing DeleteTablet for tablet acaabafbf03345b8849f0cbd0a7d4347 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:47 UTC) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:47.637751 11271 raft_consensus.cc:482] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:47.637862 11271 raft_consensus.cc:504] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:47.638067 11310 tablet_replica.cc:330] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:47.638289 11271 leader_election.cc:288] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:47.638237 11310 raft_consensus.cc:2229] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:47.638633 11310 raft_consensus.cc:2258] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:47.638626 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f403dc06195d449e8f454a9f4e795925" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:47.638774 6527 raft_consensus.cc:2454] T f403dc06195d449e8f454a9f4e795925 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:47.638810 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f403dc06195d449e8f454a9f4e795925" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:47.638939 6660 raft_consensus.cc:2454] T f403dc06195d449e8f454a9f4e795925 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:47.639340 6857 leader_election.cc:302] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:47.639495 11310 ts_tablet_manager.cc:1875] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:47.639581 11271 raft_consensus.cc:2781] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:47.639640 11271 raft_consensus.cc:482] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:47.639690 11271 raft_consensus.cc:3037] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:47.639719 11309 ts_tablet_manager.cc:1888] T acaabafbf03345b8849f0cbd0a7d4347 P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:47.639901 11309 log.cc:1196] T acaabafbf03345b8849f0cbd0a7d4347 P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/acaabafbf03345b8849f0cbd0a7d4347
I20240503 13:59:47.640270 11309 ts_tablet_manager.cc:1909] T acaabafbf03345b8849f0cbd0a7d4347 P 8d574fb4a89d4eea947c859470d44dd4: Deleting consensus metadata
I20240503 13:59:47.640708 11271 raft_consensus.cc:504] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:47.641042 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet acaabafbf03345b8849f0cbd0a7d4347 (table rebalancer_dynamic_table_new_name [id=47f3ce1454cc430db458ea8c511ea0f2]) successfully deleted
I20240503 13:59:47.641676 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f403dc06195d449e8f454a9f4e795925" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:47.641718 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f403dc06195d449e8f454a9f4e795925" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:47.641783 6660 raft_consensus.cc:3037] T f403dc06195d449e8f454a9f4e795925 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:47.641834 6527 raft_consensus.cc:3037] T f403dc06195d449e8f454a9f4e795925 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:47.641943 11310 ts_tablet_manager.cc:1888] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:47.642005 11310 log.cc:1196] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/acaabafbf03345b8849f0cbd0a7d4347
I20240503 13:59:47.642314 11310 ts_tablet_manager.cc:1909] T acaabafbf03345b8849f0cbd0a7d4347 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting consensus metadata
I20240503 13:59:47.642709 6660 raft_consensus.cc:2454] T f403dc06195d449e8f454a9f4e795925 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:47.642875 6527 raft_consensus.cc:2454] T f403dc06195d449e8f454a9f4e795925 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:47.642941 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet acaabafbf03345b8849f0cbd0a7d4347 (table rebalancer_dynamic_table_new_name [id=47f3ce1454cc430db458ea8c511ea0f2]) successfully deleted
I20240503 13:59:47.643075 6858 leader_election.cc:302] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:47.643221 11271 leader_election.cc:288] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:47.643260 11306 raft_consensus.cc:2781] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:47.643350 11306 raft_consensus.cc:686] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:47.643473 11306 consensus_queue.cc:237] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:47.645785 11308 ts_tablet_manager.cc:1888] T acaabafbf03345b8849f0cbd0a7d4347 P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:47.645860 11308 log.cc:1196] T acaabafbf03345b8849f0cbd0a7d4347 P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/acaabafbf03345b8849f0cbd0a7d4347
I20240503 13:59:47.646013 6380 catalog_manager.cc:5484] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:47.646399 11308 ts_tablet_manager.cc:1909] T acaabafbf03345b8849f0cbd0a7d4347 P 0d77752181e944698e809e325e5937e6: Deleting consensus metadata
I20240503 13:59:47.647078 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet acaabafbf03345b8849f0cbd0a7d4347 (table rebalancer_dynamic_table_new_name [id=47f3ce1454cc430db458ea8c511ea0f2]) successfully deleted
I20240503 13:59:47.736140 6380 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:47.741868 9192 tablet_service.cc:1467] Processing CreateTablet for tablet d63eac61790d47c19fdbb4ce9ea0beaa (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e0bac363bbb14ce3930187d05f48dea1]), partition=
I20240503 13:59:47.742051 6900 tablet_service.cc:1467] Processing CreateTablet for tablet d63eac61790d47c19fdbb4ce9ea0beaa (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e0bac363bbb14ce3930187d05f48dea1]), partition=
I20240503 13:59:47.742170 9192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d63eac61790d47c19fdbb4ce9ea0beaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:47.742266 6900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d63eac61790d47c19fdbb4ce9ea0beaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:47.744320 11290 tablet_bootstrap.cc:492] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:47.744395 11295 tablet_bootstrap.cc:492] T d63eac61790d47c19fdbb4ce9ea0beaa P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:47.745070 11290 tablet_bootstrap.cc:654] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:47.745114 11295 tablet_bootstrap.cc:654] T d63eac61790d47c19fdbb4ce9ea0beaa P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:47.745977 6639 tablet_service.cc:1467] Processing CreateTablet for tablet d63eac61790d47c19fdbb4ce9ea0beaa (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e0bac363bbb14ce3930187d05f48dea1]), partition=
I20240503 13:59:47.746141 11295 tablet_bootstrap.cc:492] T d63eac61790d47c19fdbb4ce9ea0beaa P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:47.746228 11295 ts_tablet_manager.cc:1367] T d63eac61790d47c19fdbb4ce9ea0beaa P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:47.746260 6639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d63eac61790d47c19fdbb4ce9ea0beaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:47.746618 11295 raft_consensus.cc:348] T d63eac61790d47c19fdbb4ce9ea0beaa P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:47.746694 11295 raft_consensus.cc:374] T d63eac61790d47c19fdbb4ce9ea0beaa P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:47.746714 11295 raft_consensus.cc:729] T d63eac61790d47c19fdbb4ce9ea0beaa P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:47.746822 11295 consensus_queue.cc:260] T d63eac61790d47c19fdbb4ce9ea0beaa P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:47.747083 11295 ts_tablet_manager.cc:1398] T d63eac61790d47c19fdbb4ce9ea0beaa P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:47.748414 11294 tablet_bootstrap.cc:492] T d63eac61790d47c19fdbb4ce9ea0beaa P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:47.748958 11290 tablet_bootstrap.cc:492] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:47.749018 11290 ts_tablet_manager.cc:1367] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240503 13:59:47.749372 11294 tablet_bootstrap.cc:654] T d63eac61790d47c19fdbb4ce9ea0beaa P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:47.749361 11290 raft_consensus.cc:348] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:47.749439 11290 raft_consensus.cc:374] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:47.749466 11290 raft_consensus.cc:729] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:47.749578 11290 consensus_queue.cc:260] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:47.749822 11290 ts_tablet_manager.cc:1398] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 13:59:47.750478 11294 tablet_bootstrap.cc:492] T d63eac61790d47c19fdbb4ce9ea0beaa P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:47.750566 11294 ts_tablet_manager.cc:1367] T d63eac61790d47c19fdbb4ce9ea0beaa P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:47.751253 11294 raft_consensus.cc:348] T d63eac61790d47c19fdbb4ce9ea0beaa P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:47.751391 11294 raft_consensus.cc:374] T d63eac61790d47c19fdbb4ce9ea0beaa P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:47.751426 11294 raft_consensus.cc:729] T d63eac61790d47c19fdbb4ce9ea0beaa P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:47.751544 11294 consensus_queue.cc:260] T d63eac61790d47c19fdbb4ce9ea0beaa P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:47.751829 11294 ts_tablet_manager.cc:1398] T d63eac61790d47c19fdbb4ce9ea0beaa P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:47.788162 11206 consensus_queue.cc:1035] T 075e3dc3a3de4a42a6a3301b821f1f3d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:47.792603 11206 consensus_queue.cc:1035] T 075e3dc3a3de4a42a6a3301b821f1f3d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:48.069587 11311 consensus_queue.cc:1035] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:48.073789 11306 consensus_queue.cc:1035] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:48.162289 6385 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:48.164675 11311 raft_consensus.cc:482] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:48.164793 11311 raft_consensus.cc:504] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:48.165500 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63eac61790d47c19fdbb4ce9ea0beaa" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:48.165633 9205 raft_consensus.cc:2454] T d63eac61790d47c19fdbb4ce9ea0beaa P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:48.165910 6857 leader_election.cc:302] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab1d21935d6049888eb0ceccc3c9d735, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:48.165969 11311 leader_election.cc:288] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:48.166154 11306 raft_consensus.cc:2781] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:48.166204 11306 raft_consensus.cc:482] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:48.166244 11306 raft_consensus.cc:3037] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:48.166883 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63eac61790d47c19fdbb4ce9ea0beaa" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:48.167012 6660 raft_consensus.cc:2454] T d63eac61790d47c19fdbb4ce9ea0beaa P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:48.167416 11306 raft_consensus.cc:504] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:48.167773 11306 leader_election.cc:288] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:48.168063 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63eac61790d47c19fdbb4ce9ea0beaa" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:48.168267 6660 raft_consensus.cc:3037] T d63eac61790d47c19fdbb4ce9ea0beaa P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:48.169332 6507 tablet_service.cc:1467] Processing CreateTablet for tablet 01551845458048e48df9901947b4eec6 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=fb05f2e903bb419db0bfe7d54f35b083]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:48.169652 6507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01551845458048e48df9901947b4eec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:48.169684 8926 tablet_service.cc:1467] Processing CreateTablet for tablet 01551845458048e48df9901947b4eec6 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=fb05f2e903bb419db0bfe7d54f35b083]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:48.169831 9321 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba45113776347c3adeae091c95427df (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=fb05f2e903bb419db0bfe7d54f35b083]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:48.169961 8926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01551845458048e48df9901947b4eec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:48.170096 9321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ba45113776347c3adeae091c95427df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:48.171833 11325 tablet_bootstrap.cc:492] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:48.172435 6771 tablet_service.cc:1467] Processing CreateTablet for tablet 01551845458048e48df9901947b4eec6 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=fb05f2e903bb419db0bfe7d54f35b083]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:48.172704 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01551845458048e48df9901947b4eec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:48.172643 6769 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba45113776347c3adeae091c95427df (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=fb05f2e903bb419db0bfe7d54f35b083]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:48.172923 11325 tablet_bootstrap.cc:654] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:48.173008 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ba45113776347c3adeae091c95427df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:48.174072 11325 tablet_bootstrap.cc:492] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:48.174163 11325 ts_tablet_manager.cc:1367] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:48.174623 11325 raft_consensus.cc:348] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:48.174742 11325 raft_consensus.cc:374] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:48.174700 6770 tablet_service.cc:1467] Processing CreateTablet for tablet e2d398cb36d54bc785648751b5b52370 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=fb05f2e903bb419db0bfe7d54f35b083]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:48.174774 11325 raft_consensus.cc:729] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:48.174882 11325 consensus_queue.cc:260] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:48.174926 6770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d398cb36d54bc785648751b5b52370. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:48.174969 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63eac61790d47c19fdbb4ce9ea0beaa" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:48.175073 9205 raft_consensus.cc:3037] T d63eac61790d47c19fdbb4ce9ea0beaa P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:48.175172 11325 ts_tablet_manager.cc:1398] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:48.175839 9205 raft_consensus.cc:2454] T d63eac61790d47c19fdbb4ce9ea0beaa P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:48.176182 6857 leader_election.cc:302] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab1d21935d6049888eb0ceccc3c9d735, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:48.176479 6508 tablet_service.cc:1467] Processing CreateTablet for tablet e2d398cb36d54bc785648751b5b52370 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=fb05f2e903bb419db0bfe7d54f35b083]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:48.176591 8926 tablet_service.cc:1467] Processing CreateTablet for tablet e2d398cb36d54bc785648751b5b52370 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=fb05f2e903bb419db0bfe7d54f35b083]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:48.176481 6507 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba45113776347c3adeae091c95427df (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=fb05f2e903bb419db0bfe7d54f35b083]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:48.176754 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d398cb36d54bc785648751b5b52370. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:48.176787 8926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d398cb36d54bc785648751b5b52370. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:48.177294 6507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ba45113776347c3adeae091c95427df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:48.177843 6660 raft_consensus.cc:2454] T d63eac61790d47c19fdbb4ce9ea0beaa P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:48.178601 11327 tablet_bootstrap.cc:492] T 6ba45113776347c3adeae091c95427df P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:48.178838 11306 raft_consensus.cc:2781] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:48.178939 11306 raft_consensus.cc:686] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:48.179083 11306 consensus_queue.cc:237] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } }
I20240503 13:59:48.179832 11327 tablet_bootstrap.cc:654] T 6ba45113776347c3adeae091c95427df P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:48.179987 11325 tablet_bootstrap.cc:492] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:48.181124 11325 tablet_bootstrap.cc:654] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:48.181479 6385 catalog_manager.cc:5484] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:48.182353 11325 tablet_bootstrap.cc:492] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545: No bootstrap required, opened a new log
I20240503 13:59:48.182435 11325 ts_tablet_manager.cc:1367] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:48.182852 11325 raft_consensus.cc:348] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:48.182943 11325 raft_consensus.cc:374] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:48.182976 11325 raft_consensus.cc:729] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: FOLLOWER
I20240503 13:59:48.183081 11325 consensus_queue.cc:260] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:48.183431 11325 ts_tablet_manager.cc:1398] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:48.184000 11329 tablet_bootstrap.cc:492] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:48.184865 11329 tablet_bootstrap.cc:654] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:48.186342 11326 tablet_bootstrap.cc:492] T 01551845458048e48df9901947b4eec6 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:48.186421 11327 tablet_bootstrap.cc:492] T 6ba45113776347c3adeae091c95427df P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:48.186561 11327 ts_tablet_manager.cc:1367] T 6ba45113776347c3adeae091c95427df P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:59:48.187023 11327 raft_consensus.cc:348] T 6ba45113776347c3adeae091c95427df P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:48.187209 11327 raft_consensus.cc:374] T 6ba45113776347c3adeae091c95427df P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:48.187332 11327 raft_consensus.cc:729] T 6ba45113776347c3adeae091c95427df P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:48.187345 11326 tablet_bootstrap.cc:654] T 01551845458048e48df9901947b4eec6 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:48.187508 11327 consensus_queue.cc:260] T 6ba45113776347c3adeae091c95427df P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:48.187856 11327 ts_tablet_manager.cc:1398] T 6ba45113776347c3adeae091c95427df P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:48.188287 11329 tablet_bootstrap.cc:492] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:48.188380 11329 ts_tablet_manager.cc:1367] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:59:48.188462 11326 tablet_bootstrap.cc:492] T 01551845458048e48df9901947b4eec6 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:48.188547 11326 ts_tablet_manager.cc:1367] T 01551845458048e48df9901947b4eec6 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:59:48.188844 11329 raft_consensus.cc:348] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:48.188944 11326 raft_consensus.cc:348] T 01551845458048e48df9901947b4eec6 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:48.189023 11329 raft_consensus.cc:374] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:48.189085 11329 raft_consensus.cc:729] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:48.189169 11326 raft_consensus.cc:374] T 01551845458048e48df9901947b4eec6 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:48.189254 11326 raft_consensus.cc:729] T 01551845458048e48df9901947b4eec6 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:48.189254 11329 consensus_queue.cc:260] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:48.189432 11326 consensus_queue.cc:260] T 01551845458048e48df9901947b4eec6 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:48.189760 11326 ts_tablet_manager.cc:1398] T 01551845458048e48df9901947b4eec6 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:48.189965 11326 tablet_bootstrap.cc:492] T e2d398cb36d54bc785648751b5b52370 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:48.190886 11326 tablet_bootstrap.cc:654] T e2d398cb36d54bc785648751b5b52370 P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:48.192070 11329 ts_tablet_manager.cc:1398] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:59:48.192279 11329 tablet_bootstrap.cc:492] T e2d398cb36d54bc785648751b5b52370 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:48.193166 11326 tablet_bootstrap.cc:492] T e2d398cb36d54bc785648751b5b52370 P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:48.193295 11326 ts_tablet_manager.cc:1367] T e2d398cb36d54bc785648751b5b52370 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:48.193495 11329 tablet_bootstrap.cc:654] T e2d398cb36d54bc785648751b5b52370 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:48.193744 11326 raft_consensus.cc:348] T e2d398cb36d54bc785648751b5b52370 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:48.193861 11326 raft_consensus.cc:374] T e2d398cb36d54bc785648751b5b52370 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:48.193892 11326 raft_consensus.cc:729] T e2d398cb36d54bc785648751b5b52370 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:48.194056 11326 consensus_queue.cc:260] T e2d398cb36d54bc785648751b5b52370 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:48.194721 11326 ts_tablet_manager.cc:1398] T e2d398cb36d54bc785648751b5b52370 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:48.194875 11326 tablet_bootstrap.cc:492] T 6ba45113776347c3adeae091c95427df P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:48.195741 11326 tablet_bootstrap.cc:654] T 6ba45113776347c3adeae091c95427df P 5f46e19644fd4f4b825c86ba6af768da: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:48.195952 11329 tablet_bootstrap.cc:492] T e2d398cb36d54bc785648751b5b52370 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:48.196058 11329 ts_tablet_manager.cc:1367] T e2d398cb36d54bc785648751b5b52370 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:48.197185 11329 raft_consensus.cc:348] T e2d398cb36d54bc785648751b5b52370 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:48.197297 11329 raft_consensus.cc:374] T e2d398cb36d54bc785648751b5b52370 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:48.197324 11329 raft_consensus.cc:729] T e2d398cb36d54bc785648751b5b52370 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:48.197466 11329 consensus_queue.cc:260] T e2d398cb36d54bc785648751b5b52370 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:48.197714 11329 ts_tablet_manager.cc:1398] T e2d398cb36d54bc785648751b5b52370 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:48.197907 11329 tablet_bootstrap.cc:492] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:48.198679 11329 tablet_bootstrap.cc:654] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:48.199635 11329 tablet_bootstrap.cc:492] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:48.199702 11326 tablet_bootstrap.cc:492] T 6ba45113776347c3adeae091c95427df P 5f46e19644fd4f4b825c86ba6af768da: No bootstrap required, opened a new log
I20240503 13:59:48.199707 11329 ts_tablet_manager.cc:1367] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:48.199759 11326 ts_tablet_manager.cc:1367] T 6ba45113776347c3adeae091c95427df P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:59:48.200049 11326 raft_consensus.cc:348] T 6ba45113776347c3adeae091c95427df P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:48.200114 11326 raft_consensus.cc:374] T 6ba45113776347c3adeae091c95427df P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:48.200134 11326 raft_consensus.cc:729] T 6ba45113776347c3adeae091c95427df P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: FOLLOWER
I20240503 13:59:48.200102 11329 raft_consensus.cc:348] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:48.200239 11326 consensus_queue.cc:260] T 6ba45113776347c3adeae091c95427df P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:48.200475 11326 ts_tablet_manager.cc:1398] T 6ba45113776347c3adeae091c95427df P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:48.200188 11329 raft_consensus.cc:374] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:48.200681 11205 raft_consensus.cc:482] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:48.201937 11329 raft_consensus.cc:729] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:48.201984 11205 raft_consensus.cc:504] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:48.202245 11329 consensus_queue.cc:260] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:48.202473 11205 leader_election.cc:288] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:48.202664 11329 ts_tablet_manager.cc:1398] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:59:48.203338 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01551845458048e48df9901947b4eec6" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 13:59:48.203492 8948 raft_consensus.cc:2454] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:48.203811 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01551845458048e48df9901947b4eec6" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:48.203958 6528 raft_consensus.cc:2454] T 01551845458048e48df9901947b4eec6 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:48.204217 6726 leader_election.cc:302] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:48.204538 11205 raft_consensus.cc:2781] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:48.204613 11205 raft_consensus.cc:482] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:48.204650 11205 raft_consensus.cc:3037] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:48.204712 11206 raft_consensus.cc:482] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:48.204790 11206 raft_consensus.cc:504] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:48.205229 11206 leader_election.cc:288] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:48.205462 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d398cb36d54bc785648751b5b52370" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:48.205544 6791 raft_consensus.cc:2454] T e2d398cb36d54bc785648751b5b52370 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:48.205721 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d398cb36d54bc785648751b5b52370" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:48.205773 8882 leader_election.cc:302] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:48.205870 6528 raft_consensus.cc:2454] T e2d398cb36d54bc785648751b5b52370 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 0.
I20240503 13:59:48.206058 11206 raft_consensus.cc:2781] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:48.206110 11206 raft_consensus.cc:482] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:48.206154 11206 raft_consensus.cc:3037] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:48.206414 11205 raft_consensus.cc:504] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:48.206728 11205 leader_election.cc:288] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:48.207207 11206 raft_consensus.cc:504] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:48.207592 11206 leader_election.cc:288] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:48.207758 8948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01551845458048e48df9901947b4eec6" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 13:59:48.207854 8948 raft_consensus.cc:3037] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:48.207904 11311 consensus_queue.cc:1035] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:48.207975 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d398cb36d54bc785648751b5b52370" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:48.208076 6527 raft_consensus.cc:3037] T e2d398cb36d54bc785648751b5b52370 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:48.208746 8948 raft_consensus.cc:2454] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:48.209040 6528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01551845458048e48df9901947b4eec6" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:48.209151 6528 raft_consensus.cc:3037] T 01551845458048e48df9901947b4eec6 P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:48.209123 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d398cb36d54bc785648751b5b52370" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:48.209219 6791 raft_consensus.cc:3037] T e2d398cb36d54bc785648751b5b52370 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:48.209903 6528 raft_consensus.cc:2454] T 01551845458048e48df9901947b4eec6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:48.210068 6791 raft_consensus.cc:2454] T e2d398cb36d54bc785648751b5b52370 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:48.210407 6726 leader_election.cc:302] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 13:59:48.210429 8882 leader_election.cc:302] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:48.209026 6527 raft_consensus.cc:2454] T e2d398cb36d54bc785648751b5b52370 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 13:59:48.210809 11206 raft_consensus.cc:2781] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:48.210908 11206 raft_consensus.cc:686] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 13:59:48.211027 11206 consensus_queue.cc:237] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:48.211409 10737 raft_consensus.cc:2781] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:48.211588 10737 raft_consensus.cc:686] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:48.211712 10737 consensus_queue.cc:237] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 13:59:48.213297 6385 catalog_manager.cc:5484] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:48.215648 6379 catalog_manager.cc:5484] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:48.218708 11311 consensus_queue.cc:1035] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:48.297716 6380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:48.301546 6658 raft_consensus.cc:1261] T d63eac61790d47c19fdbb4ce9ea0beaa P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:48.301767 9205 raft_consensus.cc:1261] T d63eac61790d47c19fdbb4ce9ea0beaa P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:48.301934 11311 consensus_queue.cc:1035] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:48.302141 11271 consensus_queue.cc:1035] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:48.305356 6380 catalog_manager.cc:5576] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091) does not have the latest schema for tablet d63eac61790d47c19fdbb4ce9ea0beaa (table rebalancer_dynamic_table [id=e0bac363bbb14ce3930187d05f48dea1]). Expected version 1 got 0
I20240503 13:59:48.309276 11350 tablet.cc:1722] T d63eac61790d47c19fdbb4ce9ea0beaa P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:48.309612 11350 tablet.cc:1620] T d63eac61790d47c19fdbb4ce9ea0beaa P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:48.309609 6380 catalog_manager.cc:5576] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119) does not have the latest schema for tablet d63eac61790d47c19fdbb4ce9ea0beaa (table rebalancer_dynamic_table [id=e0bac363bbb14ce3930187d05f48dea1]). Expected version 1 got 0
I20240503 13:59:48.315946 6380 catalog_manager.cc:5576] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531) does not have the latest schema for tablet d63eac61790d47c19fdbb4ce9ea0beaa (table rebalancer_dynamic_table [id=e0bac363bbb14ce3930187d05f48dea1]). Expected version 1 got 0
I20240503 13:59:48.316902 6379 catalog_manager.cc:5576] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119) does not have the latest schema for tablet d63eac61790d47c19fdbb4ce9ea0beaa (table rebalancer_dynamic_table [id=e0bac363bbb14ce3930187d05f48dea1]). Expected version 1 got 0
I20240503 13:59:48.316253 6385 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=e0bac363bbb14ce3930187d05f48dea1] alter complete (version 1)
I20240503 13:59:48.319408 11349 tablet.cc:1722] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:48.319736 11349 tablet.cc:1620] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460: MemRowSet was empty: no flush needed.
I20240503 13:59:48.321089 6385 catalog_manager.cc:5576] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531) does not have the latest schema for tablet d63eac61790d47c19fdbb4ce9ea0beaa (table rebalancer_dynamic_table [id=e0bac363bbb14ce3930187d05f48dea1]). Expected version 1 got 0
I20240503 13:59:48.322412 11351 tablet.cc:1722] T d63eac61790d47c19fdbb4ce9ea0beaa P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:48.322741 11351 tablet.cc:1620] T d63eac61790d47c19fdbb4ce9ea0beaa P 2145d15ab2d74c0287839266f1411a43: MemRowSet was empty: no flush needed.
I20240503 13:59:48.417729 6385 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:59:48.423008 11349 tablet.cc:1722] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:48.423398 11349 tablet.cc:1620] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460: MemRowSet was empty: no flush needed.
I20240503 13:59:48.423666 11350 tablet.cc:1722] T d63eac61790d47c19fdbb4ce9ea0beaa P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:48.424178 11350 tablet.cc:1620] T d63eac61790d47c19fdbb4ce9ea0beaa P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:48.424320 11351 tablet.cc:1722] T d63eac61790d47c19fdbb4ce9ea0beaa P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:59:48.424759 11351 tablet.cc:1620] T d63eac61790d47c19fdbb4ce9ea0beaa P 2145d15ab2d74c0287839266f1411a43: MemRowSet was empty: no flush needed.
I20240503 13:59:48.428851 6385 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=e0bac363bbb14ce3930187d05f48dea1] alter complete (version 2)
I20240503 13:59:48.532930 6385 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:59:48.538547 10737 raft_consensus.cc:482] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:48.538528 11349 tablet.cc:1722] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:48.538661 10737 raft_consensus.cc:504] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:48.539119 10737 leader_election.cc:288] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:48.539515 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba45113776347c3adeae091c95427df" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:48.539614 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba45113776347c3adeae091c95427df" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 13:59:48.539665 9343 raft_consensus.cc:2454] T 6ba45113776347c3adeae091c95427df P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:48.539753 6527 raft_consensus.cc:2454] T 6ba45113776347c3adeae091c95427df P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 0.
I20240503 13:59:48.539984 6727 leader_election.cc:302] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:48.540164 10737 raft_consensus.cc:2781] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:48.540225 10737 raft_consensus.cc:482] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:48.540268 10737 raft_consensus.cc:3037] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:48.540776 11351 tablet.cc:1722] T d63eac61790d47c19fdbb4ce9ea0beaa P 2145d15ab2d74c0287839266f1411a43: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:48.541333 10737 raft_consensus.cc:504] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:48.541661 10737 leader_election.cc:288] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:48.542132 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba45113776347c3adeae091c95427df" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:48.542246 6527 raft_consensus.cc:3037] T 6ba45113776347c3adeae091c95427df P 5f46e19644fd4f4b825c86ba6af768da [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:48.543321 6527 raft_consensus.cc:2454] T 6ba45113776347c3adeae091c95427df P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:48.543794 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba45113776347c3adeae091c95427df" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:48.543915 9343 raft_consensus.cc:3037] T 6ba45113776347c3adeae091c95427df P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:48.544900 9343 raft_consensus.cc:2454] T 6ba45113776347c3adeae091c95427df P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 1.
I20240503 13:59:48.545166 6727 leader_election.cc:302] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:48.545354 10737 raft_consensus.cc:2781] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:48.545440 10737 raft_consensus.cc:686] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:48.545590 10737 consensus_queue.cc:237] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 13:59:48.546739 11350 tablet.cc:1722] T d63eac61790d47c19fdbb4ce9ea0beaa P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:59:48.547082 6383 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table_new_name [id=e0bac363bbb14ce3930187d05f48dea1] alter complete (version 3)
I20240503 13:59:48.548545 6383 catalog_manager.cc:5484] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:48.620090 10737 consensus_queue.cc:1035] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:48.624087 11205 consensus_queue.cc:1035] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:48.644702 11352 consensus_queue.cc:1035] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:48.648159 11206 consensus_queue.cc:1035] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:48.649295 6383 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:48.649518 6383 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:59:48.656363 6383 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: Sending DeleteTablet for 3 replicas of tablet d63eac61790d47c19fdbb4ce9ea0beaa
I20240503 13:59:48.656831 9192 tablet_service.cc:1514] Processing DeleteTablet for tablet d63eac61790d47c19fdbb4ce9ea0beaa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:48 UTC) from {username='slave'} at 127.0.0.1:57960
I20240503 13:59:48.656942 6900 tablet_service.cc:1514] Processing DeleteTablet for tablet d63eac61790d47c19fdbb4ce9ea0beaa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:48 UTC) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:48.657193 11360 tablet_replica.cc:330] T d63eac61790d47c19fdbb4ce9ea0beaa P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 13:59:48.657430 6639 tablet_service.cc:1514] Processing DeleteTablet for tablet d63eac61790d47c19fdbb4ce9ea0beaa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:59:48 UTC) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:48.657750 11362 tablet_replica.cc:330] T d63eac61790d47c19fdbb4ce9ea0beaa P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:48.657792 11360 raft_consensus.cc:2229] T d63eac61790d47c19fdbb4ce9ea0beaa P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:48.657958 11360 raft_consensus.cc:2258] T d63eac61790d47c19fdbb4ce9ea0beaa P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:48.657965 11362 raft_consensus.cc:2229] T d63eac61790d47c19fdbb4ce9ea0beaa P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:48.658077 11362 raft_consensus.cc:2258] T d63eac61790d47c19fdbb4ce9ea0beaa P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:48.658582 11362 ts_tablet_manager.cc:1875] T d63eac61790d47c19fdbb4ce9ea0beaa P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:48.658757 11360 ts_tablet_manager.cc:1875] T d63eac61790d47c19fdbb4ce9ea0beaa P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:48.660523 11362 ts_tablet_manager.cc:1888] T d63eac61790d47c19fdbb4ce9ea0beaa P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:48.660591 11362 log.cc:1196] T d63eac61790d47c19fdbb4ce9ea0beaa P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/d63eac61790d47c19fdbb4ce9ea0beaa
I20240503 13:59:48.660883 11362 ts_tablet_manager.cc:1909] T d63eac61790d47c19fdbb4ce9ea0beaa P 2145d15ab2d74c0287839266f1411a43: Deleting consensus metadata
I20240503 13:59:48.661118 11360 ts_tablet_manager.cc:1888] T d63eac61790d47c19fdbb4ce9ea0beaa P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:48.661199 11360 log.cc:1196] T d63eac61790d47c19fdbb4ce9ea0beaa P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/d63eac61790d47c19fdbb4ce9ea0beaa
I20240503 13:59:48.661500 11360 ts_tablet_manager.cc:1909] T d63eac61790d47c19fdbb4ce9ea0beaa P ab1d21935d6049888eb0ceccc3c9d735: Deleting consensus metadata
I20240503 13:59:48.661664 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet d63eac61790d47c19fdbb4ce9ea0beaa (table rebalancer_dynamic_table_new_name [id=e0bac363bbb14ce3930187d05f48dea1]) successfully deleted
I20240503 13:59:48.662118 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet d63eac61790d47c19fdbb4ce9ea0beaa (table rebalancer_dynamic_table_new_name [id=e0bac363bbb14ce3930187d05f48dea1]) successfully deleted
I20240503 13:59:48.664372 11361 tablet_replica.cc:330] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:48.664739 11361 raft_consensus.cc:2229] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:48.664935 11361 raft_consensus.cc:2258] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:48.665411 11361 ts_tablet_manager.cc:1875] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:59:48.667456 11361 ts_tablet_manager.cc:1888] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:59:48.667567 11361 log.cc:1196] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/d63eac61790d47c19fdbb4ce9ea0beaa
I20240503 13:59:48.667906 11361 ts_tablet_manager.cc:1909] T d63eac61790d47c19fdbb4ce9ea0beaa P f58065db70b64ef8b1dc499aa6ae4460: Deleting consensus metadata
I20240503 13:59:48.668534 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet d63eac61790d47c19fdbb4ce9ea0beaa (table rebalancer_dynamic_table_new_name [id=e0bac363bbb14ce3930187d05f48dea1]) successfully deleted
I20240503 13:59:48.758571 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:59:48.763724 6900 tablet_service.cc:1467] Processing CreateTablet for tablet d002f7a57be04915b7b0b3dcb985f550 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c0369d036158436b8526abbc1683d235]), partition=
I20240503 13:59:48.763821 9192 tablet_service.cc:1467] Processing CreateTablet for tablet d002f7a57be04915b7b0b3dcb985f550 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c0369d036158436b8526abbc1683d235]), partition=
I20240503 13:59:48.764008 6900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d002f7a57be04915b7b0b3dcb985f550. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:48.764051 9192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d002f7a57be04915b7b0b3dcb985f550. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:48.766497 11363 tablet_bootstrap.cc:492] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:48.767051 9060 tablet_service.cc:1467] Processing CreateTablet for tablet d002f7a57be04915b7b0b3dcb985f550 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c0369d036158436b8526abbc1683d235]), partition=
I20240503 13:59:48.767338 9060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d002f7a57be04915b7b0b3dcb985f550. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:48.767482 11363 tablet_bootstrap.cc:654] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:48.768394 11364 tablet_bootstrap.cc:492] T d002f7a57be04915b7b0b3dcb985f550 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:48.769074 11363 tablet_bootstrap.cc:492] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:48.769132 11363 ts_tablet_manager.cc:1367] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:48.769399 11364 tablet_bootstrap.cc:654] T d002f7a57be04915b7b0b3dcb985f550 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:48.769461 11363 raft_consensus.cc:348] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:48.769536 11363 raft_consensus.cc:374] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:48.769555 11363 raft_consensus.cc:729] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:48.770493 11363 consensus_queue.cc:260] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:48.770597 11364 tablet_bootstrap.cc:492] T d002f7a57be04915b7b0b3dcb985f550 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:48.770695 11364 ts_tablet_manager.cc:1367] T d002f7a57be04915b7b0b3dcb985f550 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:48.770958 11363 ts_tablet_manager.cc:1398] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:48.771129 11364 raft_consensus.cc:348] T d002f7a57be04915b7b0b3dcb985f550 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:48.771237 11364 raft_consensus.cc:374] T d002f7a57be04915b7b0b3dcb985f550 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:48.771272 11364 raft_consensus.cc:729] T d002f7a57be04915b7b0b3dcb985f550 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:48.771392 11366 tablet_bootstrap.cc:492] T d002f7a57be04915b7b0b3dcb985f550 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:48.771391 11364 consensus_queue.cc:260] T d002f7a57be04915b7b0b3dcb985f550 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:48.771661 11364 ts_tablet_manager.cc:1398] T d002f7a57be04915b7b0b3dcb985f550 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:48.773412 11366 tablet_bootstrap.cc:654] T d002f7a57be04915b7b0b3dcb985f550 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:48.774749 11366 tablet_bootstrap.cc:492] T d002f7a57be04915b7b0b3dcb985f550 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:48.774842 11366 ts_tablet_manager.cc:1367] T d002f7a57be04915b7b0b3dcb985f550 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:48.775242 11366 raft_consensus.cc:348] T d002f7a57be04915b7b0b3dcb985f550 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:48.775368 11366 raft_consensus.cc:374] T d002f7a57be04915b7b0b3dcb985f550 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:48.775399 11366 raft_consensus.cc:729] T d002f7a57be04915b7b0b3dcb985f550 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:48.775549 11366 consensus_queue.cc:260] T d002f7a57be04915b7b0b3dcb985f550 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:48.775818 11366 ts_tablet_manager.cc:1398] T d002f7a57be04915b7b0b3dcb985f550 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:49.055960 6383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60286:
name: "rebalancer_extra_table_74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:49.057636 11311 raft_consensus.cc:482] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:49.057790 11311 raft_consensus.cc:504] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:49.058176 11311 leader_election.cc:288] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:49.058494 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d002f7a57be04915b7b0b3dcb985f550" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:49.058625 9081 raft_consensus.cc:2454] T d002f7a57be04915b7b0b3dcb985f550 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:49.058602 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d002f7a57be04915b7b0b3dcb985f550" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:49.058732 9211 raft_consensus.cc:2454] T d002f7a57be04915b7b0b3dcb985f550 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:49.058871 6857 leader_election.cc:302] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:49.059226 11306 raft_consensus.cc:2781] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:49.059314 11306 raft_consensus.cc:482] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:49.059360 11306 raft_consensus.cc:3037] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.060179 11306 raft_consensus.cc:504] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:49.060417 11306 leader_election.cc:288] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:49.061038 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d002f7a57be04915b7b0b3dcb985f550" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:49.061152 9211 raft_consensus.cc:3037] T d002f7a57be04915b7b0b3dcb985f550 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.062171 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d002f7a57be04915b7b0b3dcb985f550" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:49.062294 9081 raft_consensus.cc:3037] T d002f7a57be04915b7b0b3dcb985f550 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.063117 6900 tablet_service.cc:1467] Processing CreateTablet for tablet 10dd65668d544a7fb8d2edc737c303ae (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=a17de5eddcbb412081dd0148ee6471c6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:49.063364 9081 raft_consensus.cc:2454] T d002f7a57be04915b7b0b3dcb985f550 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:49.063436 6900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10dd65668d544a7fb8d2edc737c303ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.063710 9060 tablet_service.cc:1467] Processing CreateTablet for tablet a60d453dd9c04ffaa6af6a6e233c7ef2 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=a17de5eddcbb412081dd0148ee6471c6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:49.063949 9060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a60d453dd9c04ffaa6af6a6e233c7ef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.064100 9192 tablet_service.cc:1467] Processing CreateTablet for tablet 10dd65668d544a7fb8d2edc737c303ae (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=a17de5eddcbb412081dd0148ee6471c6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:49.064358 9192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10dd65668d544a7fb8d2edc737c303ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.065515 11366 tablet_bootstrap.cc:492] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:49.065585 11363 tablet_bootstrap.cc:492] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:49.065716 6769 tablet_service.cc:1467] Processing CreateTablet for tablet 5c7b12e4dc3943a299fc8a49cbc02694 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=a17de5eddcbb412081dd0148ee6471c6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:49.065990 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c7b12e4dc3943a299fc8a49cbc02694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.066188 9211 raft_consensus.cc:2454] T d002f7a57be04915b7b0b3dcb985f550 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:49.066531 11363 tablet_bootstrap.cc:654] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.067008 11366 tablet_bootstrap.cc:654] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 8d574fb4a89d4eea947c859470d44dd4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.067557 6857 leader_election.cc:302] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab1d21935d6049888eb0ceccc3c9d735, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:49.067603 9322 tablet_service.cc:1467] Processing CreateTablet for tablet a60d453dd9c04ffaa6af6a6e233c7ef2 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=a17de5eddcbb412081dd0148ee6471c6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:49.067790 9321 tablet_service.cc:1467] Processing CreateTablet for tablet 10dd65668d544a7fb8d2edc737c303ae (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=a17de5eddcbb412081dd0148ee6471c6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:49.067914 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a60d453dd9c04ffaa6af6a6e233c7ef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.067961 11311 raft_consensus.cc:2781] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:49.068023 9321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10dd65668d544a7fb8d2edc737c303ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.068028 11311 raft_consensus.cc:686] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:49.068142 11311 consensus_queue.cc:237] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 13:59:49.068508 6639 tablet_service.cc:1467] Processing CreateTablet for tablet a60d453dd9c04ffaa6af6a6e233c7ef2 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=a17de5eddcbb412081dd0148ee6471c6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:49.068754 6639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a60d453dd9c04ffaa6af6a6e233c7ef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.069586 11363 tablet_bootstrap.cc:492] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:49.069664 11363 ts_tablet_manager.cc:1367] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:49.070086 11363 raft_consensus.cc:348] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:49.070176 11363 raft_consensus.cc:374] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.070206 11363 raft_consensus.cc:729] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.070322 11363 consensus_queue.cc:260] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:49.070457 6900 tablet_service.cc:1467] Processing CreateTablet for tablet 5c7b12e4dc3943a299fc8a49cbc02694 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=a17de5eddcbb412081dd0148ee6471c6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:49.070603 11363 ts_tablet_manager.cc:1398] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:49.070658 6900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c7b12e4dc3943a299fc8a49cbc02694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.071117 11366 tablet_bootstrap.cc:492] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 8d574fb4a89d4eea947c859470d44dd4: No bootstrap required, opened a new log
I20240503 13:59:49.071192 11366 ts_tablet_manager.cc:1367] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:59:49.071609 11366 raft_consensus.cc:348] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:49.071725 11366 raft_consensus.cc:374] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.071755 11366 raft_consensus.cc:729] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.071866 11366 consensus_queue.cc:260] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:49.072147 11366 ts_tablet_manager.cc:1398] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:49.072494 11364 tablet_bootstrap.cc:492] T 10dd65668d544a7fb8d2edc737c303ae P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:49.073448 11364 tablet_bootstrap.cc:654] T 10dd65668d544a7fb8d2edc737c303ae P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.073566 6385 catalog_manager.cc:5484] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:49.074266 9192 tablet_service.cc:1467] Processing CreateTablet for tablet 5c7b12e4dc3943a299fc8a49cbc02694 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=a17de5eddcbb412081dd0148ee6471c6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:49.074523 9192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c7b12e4dc3943a299fc8a49cbc02694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.076017 11376 tablet_bootstrap.cc:492] T 10dd65668d544a7fb8d2edc737c303ae P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:49.076787 11363 tablet_bootstrap.cc:492] T 5c7b12e4dc3943a299fc8a49cbc02694 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:49.077039 11376 tablet_bootstrap.cc:654] T 10dd65668d544a7fb8d2edc737c303ae P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.077780 11363 tablet_bootstrap.cc:654] T 5c7b12e4dc3943a299fc8a49cbc02694 P f58065db70b64ef8b1dc499aa6ae4460: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.079509 11340 consensus_queue.cc:1035] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:49.079895 11364 tablet_bootstrap.cc:492] T 10dd65668d544a7fb8d2edc737c303ae P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:49.079988 11364 ts_tablet_manager.cc:1367] T 10dd65668d544a7fb8d2edc737c303ae P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240503 13:59:49.080433 11364 raft_consensus.cc:348] T 10dd65668d544a7fb8d2edc737c303ae P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:49.080533 11364 raft_consensus.cc:374] T 10dd65668d544a7fb8d2edc737c303ae P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.080562 11364 raft_consensus.cc:729] T 10dd65668d544a7fb8d2edc737c303ae P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.080669 11364 consensus_queue.cc:260] T 10dd65668d544a7fb8d2edc737c303ae P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:49.080890 11363 tablet_bootstrap.cc:492] T 5c7b12e4dc3943a299fc8a49cbc02694 P f58065db70b64ef8b1dc499aa6ae4460: No bootstrap required, opened a new log
I20240503 13:59:49.080948 11364 ts_tablet_manager.cc:1398] T 10dd65668d544a7fb8d2edc737c303ae P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:49.080965 11363 ts_tablet_manager.cc:1367] T 5c7b12e4dc3943a299fc8a49cbc02694 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:49.081092 11364 tablet_bootstrap.cc:492] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:49.081648 11377 tablet_bootstrap.cc:492] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:49.081928 11364 tablet_bootstrap.cc:654] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.082589 11377 tablet_bootstrap.cc:654] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.082789 11363 raft_consensus.cc:348] T 5c7b12e4dc3943a299fc8a49cbc02694 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:49.082908 11363 raft_consensus.cc:374] T 5c7b12e4dc3943a299fc8a49cbc02694 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.082945 11363 raft_consensus.cc:729] T 5c7b12e4dc3943a299fc8a49cbc02694 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.083081 11363 consensus_queue.cc:260] T 5c7b12e4dc3943a299fc8a49cbc02694 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:49.083350 11374 tablet_bootstrap.cc:492] T 5c7b12e4dc3943a299fc8a49cbc02694 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:49.085047 11374 tablet_bootstrap.cc:654] T 5c7b12e4dc3943a299fc8a49cbc02694 P 4a85e492c8154e9881a9cc2a105d7e64: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.085572 11364 tablet_bootstrap.cc:492] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735: No bootstrap required, opened a new log
I20240503 13:59:49.085840 11364 ts_tablet_manager.cc:1367] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 13:59:49.086289 11364 raft_consensus.cc:348] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:49.086453 11364 raft_consensus.cc:374] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.086530 11364 raft_consensus.cc:729] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.086555 11363 ts_tablet_manager.cc:1398] T 5c7b12e4dc3943a299fc8a49cbc02694 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 13:59:49.087085 11377 tablet_bootstrap.cc:492] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43: No bootstrap required, opened a new log
I20240503 13:59:49.087172 11377 ts_tablet_manager.cc:1367] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:59:49.086751 11364 consensus_queue.cc:260] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:49.087726 11377 raft_consensus.cc:348] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:49.087846 11377 raft_consensus.cc:374] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.087884 11377 raft_consensus.cc:729] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.088058 11377 consensus_queue.cc:260] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:49.088478 11377 ts_tablet_manager.cc:1398] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:49.088625 11306 raft_consensus.cc:482] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:49.088708 11306 raft_consensus.cc:504] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:49.089110 11306 leader_election.cc:288] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:49.089618 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10dd65668d544a7fb8d2edc737c303ae" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6" is_pre_election: true
I20240503 13:59:49.090107 10737 consensus_queue.cc:1035] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 13:59:49.090181 6858 leader_election.cc:341] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:59:49.090421 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10dd65668d544a7fb8d2edc737c303ae" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:49.090552 9211 raft_consensus.cc:2454] T 10dd65668d544a7fb8d2edc737c303ae P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 0.
I20240503 13:59:49.090831 6857 leader_election.cc:302] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab1d21935d6049888eb0ceccc3c9d735, f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6
I20240503 13:59:49.091053 11306 raft_consensus.cc:2781] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:49.091100 11306 raft_consensus.cc:482] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:49.091151 11306 raft_consensus.cc:3037] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.091912 11374 tablet_bootstrap.cc:492] T 5c7b12e4dc3943a299fc8a49cbc02694 P 4a85e492c8154e9881a9cc2a105d7e64: No bootstrap required, opened a new log
I20240503 13:59:49.092132 11374 ts_tablet_manager.cc:1367] T 5c7b12e4dc3943a299fc8a49cbc02694 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 13:59:49.092190 11306 raft_consensus.cc:504] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:49.092532 11306 leader_election.cc:288] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:49.092664 11374 raft_consensus.cc:348] T 5c7b12e4dc3943a299fc8a49cbc02694 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:49.092850 11374 raft_consensus.cc:374] T 5c7b12e4dc3943a299fc8a49cbc02694 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.093173 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10dd65668d544a7fb8d2edc737c303ae" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:49.093194 9342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10dd65668d544a7fb8d2edc737c303ae" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:49.093295 9211 raft_consensus.cc:3037] T 10dd65668d544a7fb8d2edc737c303ae P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.093408 11364 ts_tablet_manager.cc:1398] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 13:59:49.094245 9211 raft_consensus.cc:2454] T 10dd65668d544a7fb8d2edc737c303ae P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 13:59:49.094717 6857 leader_election.cc:302] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab1d21935d6049888eb0ceccc3c9d735, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:49.094825 11376 tablet_bootstrap.cc:492] T 10dd65668d544a7fb8d2edc737c303ae P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:49.092962 11374 raft_consensus.cc:729] T 5c7b12e4dc3943a299fc8a49cbc02694 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.095016 11374 consensus_queue.cc:260] T 5c7b12e4dc3943a299fc8a49cbc02694 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
W20240503 13:59:49.095074 6858 leader_election.cc:341] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:59:49.095181 11376 ts_tablet_manager.cc:1367] T 10dd65668d544a7fb8d2edc737c303ae P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 13:59:49.095674 11376 raft_consensus.cc:348] T 10dd65668d544a7fb8d2edc737c303ae P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:49.095840 11376 raft_consensus.cc:374] T 10dd65668d544a7fb8d2edc737c303ae P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.095918 11376 raft_consensus.cc:729] T 10dd65668d544a7fb8d2edc737c303ae P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.096232 11306 raft_consensus.cc:2781] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:49.096315 11306 raft_consensus.cc:686] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:49.096068 11376 consensus_queue.cc:260] T 10dd65668d544a7fb8d2edc737c303ae P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:49.096441 11306 consensus_queue.cc:237] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:49.096691 11376 ts_tablet_manager.cc:1398] T 10dd65668d544a7fb8d2edc737c303ae P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:49.096851 11376 tablet_bootstrap.cc:492] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:49.097793 11376 tablet_bootstrap.cc:654] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 0d77752181e944698e809e325e5937e6: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.099139 11374 ts_tablet_manager.cc:1398] T 5c7b12e4dc3943a299fc8a49cbc02694 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 13:59:49.101634 11376 tablet_bootstrap.cc:492] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 0d77752181e944698e809e325e5937e6: No bootstrap required, opened a new log
I20240503 13:59:49.101729 11376 ts_tablet_manager.cc:1367] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 13:59:49.102418 11376 raft_consensus.cc:348] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:49.102521 11376 raft_consensus.cc:374] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.102558 11376 raft_consensus.cc:729] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.102667 11376 consensus_queue.cc:260] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:49.102928 11376 ts_tablet_manager.cc:1398] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:49.105579 6383 catalog_manager.cc:5484] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:49.179823 6385 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60286:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:59:49.184800 9081 raft_consensus.cc:1261] T d002f7a57be04915b7b0b3dcb985f550 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:49.184808 9211 raft_consensus.cc:1261] T d002f7a57be04915b7b0b3dcb985f550 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:49.185249 11271 consensus_queue.cc:1035] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:49.185540 11271 consensus_queue.cc:1035] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:49.187880 11395 tablet.cc:1722] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:49.188174 11395 tablet.cc:1620] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460: MemRowSet was empty: no flush needed.
I20240503 13:59:49.189246 6385 catalog_manager.cc:5576] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481) does not have the latest schema for tablet d002f7a57be04915b7b0b3dcb985f550 (table rebalancer_dynamic_table [id=c0369d036158436b8526abbc1683d235]). Expected version 1 got 0
I20240503 13:59:49.189687 6386 catalog_manager.cc:5576] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531) does not have the latest schema for tablet d002f7a57be04915b7b0b3dcb985f550 (table rebalancer_dynamic_table [id=c0369d036158436b8526abbc1683d235]). Expected version 1 got 0
I20240503 13:59:49.190793 6383 catalog_manager.cc:5576] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091) does not have the latest schema for tablet d002f7a57be04915b7b0b3dcb985f550 (table rebalancer_dynamic_table [id=c0369d036158436b8526abbc1683d235]). Expected version 1 got 0
I20240503 13:59:49.193588 6383 catalog_manager.cc:5576] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481) does not have the latest schema for tablet d002f7a57be04915b7b0b3dcb985f550 (table rebalancer_dynamic_table [id=c0369d036158436b8526abbc1683d235]). Expected version 1 got 0
I20240503 13:59:49.195142 11396 tablet.cc:1722] T d002f7a57be04915b7b0b3dcb985f550 P 8d574fb4a89d4eea947c859470d44dd4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:49.195603 11396 tablet.cc:1620] T d002f7a57be04915b7b0b3dcb985f550 P 8d574fb4a89d4eea947c859470d44dd4: MemRowSet was empty: no flush needed.
I20240503 13:59:49.197360 6383 catalog_manager.cc:5576] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091) does not have the latest schema for tablet d002f7a57be04915b7b0b3dcb985f550 (table rebalancer_dynamic_table [id=c0369d036158436b8526abbc1683d235]). Expected version 1 got 0
I20240503 13:59:49.197566 11397 tablet.cc:1722] T d002f7a57be04915b7b0b3dcb985f550 P ab1d21935d6049888eb0ceccc3c9d735: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:49.197489 6386 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 453655c40fe0450b838a8fa099757ca4: rebalancer_dynamic_table [id=c0369d036158436b8526abbc1683d235] alter complete (version 1)
I20240503 13:59:49.199505 11397 tablet.cc:1620] T d002f7a57be04915b7b0b3dcb985f550 P ab1d21935d6049888eb0ceccc3c9d735: MemRowSet was empty: no flush needed.
I20240503 13:59:49.220746 11378 raft_consensus.cc:482] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:49.220875 11378 raft_consensus.cc:504] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:49.221135 11260 raft_consensus.cc:482] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:49.221256 11260 raft_consensus.cc:504] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:49.221395 11378 leader_election.cc:288] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:49.221616 11260 leader_election.cc:288] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:49.221634 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60d453dd9c04ffaa6af6a6e233c7ef2" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 13:59:49.221698 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60d453dd9c04ffaa6af6a6e233c7ef2" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 13:59:49.221784 9081 raft_consensus.cc:2454] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:49.221855 6660 raft_consensus.cc:2454] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 0.
I20240503 13:59:49.222064 9278 leader_election.cc:302] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:49.222218 11378 raft_consensus.cc:2781] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:49.222276 11378 raft_consensus.cc:482] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:49.222257 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c7b12e4dc3943a299fc8a49cbc02694" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:49.222319 11378 raft_consensus.cc:3037] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.222280 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c7b12e4dc3943a299fc8a49cbc02694" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64" is_pre_election: true
I20240503 13:59:49.222381 6922 raft_consensus.cc:2454] T 5c7b12e4dc3943a299fc8a49cbc02694 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:49.222409 6791 raft_consensus.cc:2454] T 5c7b12e4dc3943a299fc8a49cbc02694 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 0.
I20240503 13:59:49.222656 9147 leader_election.cc:302] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab1d21935d6049888eb0ceccc3c9d735, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:49.222827 11260 raft_consensus.cc:2781] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:49.222872 11260 raft_consensus.cc:482] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:49.222906 11260 raft_consensus.cc:3037] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.223546 11378 raft_consensus.cc:504] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:49.223903 11378 leader_election.cc:288] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:49.223915 11260 raft_consensus.cc:504] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:49.224189 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60d453dd9c04ffaa6af6a6e233c7ef2" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:49.224280 11260 leader_election.cc:288] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:49.224303 9081 raft_consensus.cc:3037] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 8d574fb4a89d4eea947c859470d44dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.224658 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c7b12e4dc3943a299fc8a49cbc02694" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:49.224777 6922 raft_consensus.cc:3037] T 5c7b12e4dc3943a299fc8a49cbc02694 P f58065db70b64ef8b1dc499aa6ae4460 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.224833 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60d453dd9c04ffaa6af6a6e233c7ef2" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:49.224933 6660 raft_consensus.cc:3037] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.225327 9081 raft_consensus.cc:2454] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:49.225651 6922 raft_consensus.cc:2454] T 5c7b12e4dc3943a299fc8a49cbc02694 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:49.225723 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c7b12e4dc3943a299fc8a49cbc02694" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:49.225854 6791 raft_consensus.cc:3037] T 5c7b12e4dc3943a299fc8a49cbc02694 P 4a85e492c8154e9881a9cc2a105d7e64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.225973 6660 raft_consensus.cc:2454] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77752181e944698e809e325e5937e6 in term 1.
I20240503 13:59:49.226352 9278 leader_election.cc:302] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:49.226512 11378 raft_consensus.cc:2781] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:49.226601 11378 raft_consensus.cc:686] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:49.226735 11378 consensus_queue.cc:237] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:49.226959 6791 raft_consensus.cc:2454] T 5c7b12e4dc3943a299fc8a49cbc02694 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 13:59:49.227306 9146 leader_election.cc:302] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:49.228904 11260 raft_consensus.cc:2781] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:49.229000 11260 raft_consensus.cc:686] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 13:59:49.229126 11260 consensus_queue.cc:237] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 13:59:49.230569 6386 catalog_manager.cc:5484] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 0d77752181e944698e809e325e5937e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77752181e944698e809e325e5937e6 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:49.231418 6383 catalog_manager.cc:5484] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:49.306491 11378 raft_consensus.cc:482] T 10dd65668d544a7fb8d2edc737c303ae P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:49.306604 11378 raft_consensus.cc:504] T 10dd65668d544a7fb8d2edc737c303ae P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 13:59:49.307035 11378 leader_election.cc:288] T 10dd65668d544a7fb8d2edc737c303ae P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:49.307439 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10dd65668d544a7fb8d2edc737c303ae" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 13:59:49.307503 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10dd65668d544a7fb8d2edc737c303ae" candidate_uuid: "0d77752181e944698e809e325e5937e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 13:59:49.307649 9211 raft_consensus.cc:2379] T 10dd65668d544a7fb8d2edc737c303ae P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d77752181e944698e809e325e5937e6 in current term 1: Already voted for candidate f58065db70b64ef8b1dc499aa6ae4460 in this term.
I20240503 13:59:49.308070 9278 leader_election.cc:302] T 10dd65668d544a7fb8d2edc737c303ae P 0d77752181e944698e809e325e5937e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d77752181e944698e809e325e5937e6; no voters: ab1d21935d6049888eb0ceccc3c9d735, f58065db70b64ef8b1dc499aa6ae4460
I20240503 13:59:49.308247 11378 raft_consensus.cc:3037] T 10dd65668d544a7fb8d2edc737c303ae P 0d77752181e944698e809e325e5937e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.309432 11378 raft_consensus.cc:2726] T 10dd65668d544a7fb8d2edc737c303ae P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 13:59:49.609942 8926 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 13:59:49.614044 6769 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:49.621659 9321 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240503 13:59:49.625187 6507 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:59:49.630070 6639 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 13:59:49.644711 11271 consensus_queue.cc:1035] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:49.647960 9192 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:49.658859 9060 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:59:49.664376 11306 consensus_queue.cc:1035] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:49.685799 11167 consensus_queue.cc:1035] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:49.686769 6900 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 13:59:49.707903 11378 consensus_queue.cc:1035] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:49.715384 11370 consensus_queue.cc:1035] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:49.718154 11370 consensus_queue.cc:1035] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:49.923472 9321 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240503 13:59:49.926373 8926 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 13:59:49.935617 6769 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:49.942384 9060 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:59:49.942986 6639 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 13:59:49.947484 9192 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:49.979774 6900 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 13:59:49.980108 6507 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:59:50.242668 9321 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240503 13:59:50.244797 6639 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 13:59:50.246960 6769 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:50.247565 6507 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:59:50.269701 9060 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:59:50.270730 9192 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:50.274049 6900 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 13:59:50.310626 8926 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 13:59:50.406301 9343 consensus_queue.cc:237] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:50.407687 8948 raft_consensus.cc:1261] T 0fa4727297da40eb9f58f5daa8150e0e P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:50.408175 9279 consensus_peers.cc:485] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 -> Peer 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): Couldn't send request to peer 2145d15ab2d74c0287839266f1411a43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0fa4727297da40eb9f58f5daa8150e0e. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:50.408239 6527 raft_consensus.cc:1261] T 0fa4727297da40eb9f58f5daa8150e0e P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:50.408468 11167 consensus_queue.cc:1035] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.408813 11167 consensus_queue.cc:1035] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.410259 11172 raft_consensus.cc:2932] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:50.410640 8948 raft_consensus.cc:2932] T 0fa4727297da40eb9f58f5daa8150e0e P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:50.413367 6527 raft_consensus.cc:2932] T 0fa4727297da40eb9f58f5daa8150e0e P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:50.416288 6381 catalog_manager.cc:5484] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:50.423854 6658 consensus_queue.cc:237] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:50.425523 6922 raft_consensus.cc:1261] T 8f57f1ad5c604b5e957374bab029a5df P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:50.425583 6790 raft_consensus.cc:1261] T 8f57f1ad5c604b5e957374bab029a5df P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:59:50.425645 6595 consensus_peers.cc:485] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 -> Peer ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): Couldn't send request to peer ab1d21935d6049888eb0ceccc3c9d735. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f57f1ad5c604b5e957374bab029a5df. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:50.425951 11603 consensus_queue.cc:1035] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:50.426123 11584 consensus_queue.cc:1035] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:50.427815 11603 raft_consensus.cc:2932] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:50.428249 6920 raft_consensus.cc:2932] T 8f57f1ad5c604b5e957374bab029a5df P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:50.430661 6790 raft_consensus.cc:2932] T 8f57f1ad5c604b5e957374bab029a5df P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:50.431516 6387 catalog_manager.cc:5484] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index -1 to 4, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:50.441363 9343 consensus_queue.cc:237] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:50.442647 6527 raft_consensus.cc:1261] T 1350225b1a2848debca68e1bad5528c5 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:50.442741 6920 raft_consensus.cc:1261] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:50.442993 11167 consensus_queue.cc:1035] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.443166 11378 consensus_queue.cc:1035] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 13:59:50.443310 9277 consensus_peers.cc:485] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1350225b1a2848debca68e1bad5528c5. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:50.444965 11172 raft_consensus.cc:2932] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:50.445641 6527 raft_consensus.cc:2932] T 1350225b1a2848debca68e1bad5528c5 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:50.445659 6920 raft_consensus.cc:2932] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:50.448343 6384 catalog_manager.cc:5484] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:50.463379 9343 consensus_queue.cc:237] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:50.465132 6790 raft_consensus.cc:1261] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:50.465245 9278 consensus_peers.cc:485] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 -> Peer 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): Couldn't send request to peer 8d574fb4a89d4eea947c859470d44dd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96ce36eb7c25413eaae380f32b465df7. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:50.465508 6658 raft_consensus.cc:1261] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:50.465516 11378 consensus_queue.cc:1035] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.465873 11378 consensus_queue.cc:1035] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.467262 11378 raft_consensus.cc:2932] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:50.468655 6658 raft_consensus.cc:2932] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:50.469853 6790 raft_consensus.cc:2932] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:50.471179 6381 catalog_manager.cc:5484] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index -1 to 2, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:50.481344 9343 consensus_queue.cc:237] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:50.482786 9211 raft_consensus.cc:1261] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:50.482870 8946 raft_consensus.cc:1261] T 1eec99c0bb994dd7b6ae63918125bccb P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:50.482975 9279 consensus_peers.cc:485] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 -> Peer 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): Couldn't send request to peer 2145d15ab2d74c0287839266f1411a43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1eec99c0bb994dd7b6ae63918125bccb. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:50.483228 11172 consensus_queue.cc:1035] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.483402 11167 consensus_queue.cc:1035] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.485523 11167 raft_consensus.cc:2932] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:50.487449 8946 raft_consensus.cc:2932] T 1eec99c0bb994dd7b6ae63918125bccb P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:50.487733 9211 raft_consensus.cc:2932] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:50.488826 6384 catalog_manager.cc:5484] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:50.499970 6920 consensus_queue.cc:237] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:50.501533 6660 raft_consensus.cc:1261] T 350ba659df804b95b0692091dc2f68a2 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:50.501716 6790 raft_consensus.cc:1261] T 350ba659df804b95b0692091dc2f68a2 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:50.502023 11311 consensus_queue.cc:1035] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:50.502192 11306 consensus_queue.cc:1035] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:59:50.502297 6857 consensus_peers.cc:485] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): Couldn't send request to peer ab1d21935d6049888eb0ceccc3c9d735. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 350ba659df804b95b0692091dc2f68a2. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:50.503974 11306 raft_consensus.cc:2932] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:50.506064 6790 raft_consensus.cc:2932] T 350ba659df804b95b0692091dc2f68a2 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:50.506206 6660 raft_consensus.cc:2932] T 350ba659df804b95b0692091dc2f68a2 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:50.508334 6381 catalog_manager.cc:5484] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index -1 to 4, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:50.519017 9205 consensus_queue.cc:237] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:50.520627 8946 raft_consensus.cc:1261] T a7e9c80dbf8c41469e54535c20648df4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:50.520730 6527 raft_consensus.cc:1261] T a7e9c80dbf8c41469e54535c20648df4 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:50.520726 9146 consensus_peers.cc:485] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7e9c80dbf8c41469e54535c20648df4. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:50.521044 11370 consensus_queue.cc:1035] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.521216 11471 consensus_queue.cc:1035] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.522917 11260 raft_consensus.cc:2932] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:50.523365 8946 raft_consensus.cc:2932] T a7e9c80dbf8c41469e54535c20648df4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:50.523454 6527 raft_consensus.cc:2932] T a7e9c80dbf8c41469e54535c20648df4 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:50.526872 6387 catalog_manager.cc:5484] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:50.537526 6920 consensus_queue.cc:237] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:50.538937 9080 raft_consensus.cc:1261] T dfa48440020d4754be0f5ddf159c23fe P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:50.539005 6660 raft_consensus.cc:1261] T dfa48440020d4754be0f5ddf159c23fe P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:50.539362 6858 consensus_peers.cc:485] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dfa48440020d4754be0f5ddf159c23fe. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:50.539387 11311 consensus_queue.cc:1035] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.539542 11306 consensus_queue.cc:1035] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.541388 11271 raft_consensus.cc:2932] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:50.541755 9080 raft_consensus.cc:2932] T dfa48440020d4754be0f5ddf159c23fe P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:50.544287 6660 raft_consensus.cc:2932] T dfa48440020d4754be0f5ddf159c23fe P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:50.545246 6387 catalog_manager.cc:5484] T dfa48440020d4754be0f5ddf159c23fe P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index -1 to 2, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:50.554433 9205 consensus_queue.cc:237] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:50.555922 6527 raft_consensus.cc:1261] T f853508f20d4431bb35e3e05b659953a P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:50.555994 8946 raft_consensus.cc:1261] T f853508f20d4431bb35e3e05b659953a P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:50.556195 9148 consensus_peers.cc:485] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): Couldn't send request to peer 2145d15ab2d74c0287839266f1411a43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f853508f20d4431bb35e3e05b659953a. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:50.556452 11370 consensus_queue.cc:1035] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.556605 11471 consensus_queue.cc:1035] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.558568 11471 raft_consensus.cc:2932] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:50.558992 6527 raft_consensus.cc:2932] T f853508f20d4431bb35e3e05b659953a P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:50.560631 8946 raft_consensus.cc:2932] T f853508f20d4431bb35e3e05b659953a P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:50.562476 6381 catalog_manager.cc:5484] T f853508f20d4431bb35e3e05b659953a P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:50.572837 9080 consensus_queue.cc:237] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:50.575213 9205 raft_consensus.cc:1261] T ee688e32b7d049509a39259d13439cf6 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:50.575729 9016 consensus_peers.cc:485] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): Couldn't send request to peer f58065db70b64ef8b1dc499aa6ae4460. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee688e32b7d049509a39259d13439cf6. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:50.575847 11371 consensus_queue.cc:1035] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.575979 9343 raft_consensus.cc:1261] T ee688e32b7d049509a39259d13439cf6 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:50.576308 11371 consensus_queue.cc:1035] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.577595 11583 raft_consensus.cc:2932] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:50.578382 9205 raft_consensus.cc:2932] T ee688e32b7d049509a39259d13439cf6 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:50.580302 9343 raft_consensus.cc:2932] T ee688e32b7d049509a39259d13439cf6 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:50.582445 6384 catalog_manager.cc:5484] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:50.591261 6660 consensus_queue.cc:237] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:50.592974 6790 raft_consensus.cc:1261] T 8f9149a9faa746cd9542aab02a8e2850 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:50.593189 6595 consensus_peers.cc:485] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 -> Peer 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): Couldn't send request to peer 5f46e19644fd4f4b825c86ba6af768da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f9149a9faa746cd9542aab02a8e2850. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:50.593521 9080 raft_consensus.cc:1261] T 8f9149a9faa746cd9542aab02a8e2850 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:50.593561 11320 consensus_queue.cc:1035] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.593932 11584 consensus_queue.cc:1035] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.595702 11320 raft_consensus.cc:2932] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:50.596259 9080 raft_consensus.cc:2932] T 8f9149a9faa746cd9542aab02a8e2850 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:50.598059 6790 raft_consensus.cc:2932] T 8f9149a9faa746cd9542aab02a8e2850 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:50.598734 6384 catalog_manager.cc:5484] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:50.610031 9343 consensus_queue.cc:237] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
W20240503 13:59:50.614051 9278 consensus_peers.cc:485] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 -> Peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): Couldn't send request to peer 06ca8dbc004c42b99faa527a4e6f9545. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0fe53d65ab4d482aad118d6e3bd5533c. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:50.614156 9205 raft_consensus.cc:1261] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:50.614491 9080 raft_consensus.cc:1261] T 0fe53d65ab4d482aad118d6e3bd5533c P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:50.614573 11378 consensus_queue.cc:1035] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.614909 11378 consensus_queue.cc:1035] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.616295 11167 raft_consensus.cc:2932] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:50.616724 9205 raft_consensus.cc:2932] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:50.619695 9080 raft_consensus.cc:2932] T 0fe53d65ab4d482aad118d6e3bd5533c P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:50.619745 6387 catalog_manager.cc:5484] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index -1 to 2, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:50.629969 9205 consensus_queue.cc:237] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:50.631213 6790 raft_consensus.cc:1261] T b7e1060596204c60882388e984986ac6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:50.631439 9343 raft_consensus.cc:1261] T b7e1060596204c60882388e984986ac6 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:50.631453 9148 consensus_peers.cc:485] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): Couldn't send request to peer 2145d15ab2d74c0287839266f1411a43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b7e1060596204c60882388e984986ac6. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:50.631596 11471 consensus_queue.cc:1035] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.631786 11665 consensus_queue.cc:1035] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.633571 11370 raft_consensus.cc:2932] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:50.633971 6790 raft_consensus.cc:2932] T b7e1060596204c60882388e984986ac6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:50.636184 9343 raft_consensus.cc:2932] T b7e1060596204c60882388e984986ac6 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:50.637305 6381 catalog_manager.cc:5484] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:50.650067 6920 consensus_queue.cc:237] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:50.651325 9205 raft_consensus.cc:1261] T 7a44f5349cc44be69e9b52f8047d2bb1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:50.651643 9343 raft_consensus.cc:1261] T 7a44f5349cc44be69e9b52f8047d2bb1 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:50.651767 11654 consensus_queue.cc:1035] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 13:59:50.651968 6856 consensus_peers.cc:485] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a44f5349cc44be69e9b52f8047d2bb1. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:50.652439 11654 consensus_queue.cc:1035] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.653451 11655 raft_consensus.cc:2932] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:50.653832 9205 raft_consensus.cc:2932] T 7a44f5349cc44be69e9b52f8047d2bb1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:50.656971 6384 catalog_manager.cc:5484] T 7a44f5349cc44be69e9b52f8047d2bb1 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:50.659619 9343 raft_consensus.cc:2932] T 7a44f5349cc44be69e9b52f8047d2bb1 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:50.668941 9205 consensus_queue.cc:237] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:50.670073 6790 raft_consensus.cc:1261] T 5c7b12e4dc3943a299fc8a49cbc02694 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:50.670421 9147 consensus_peers.cc:485] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): Couldn't send request to peer 5f46e19644fd4f4b825c86ba6af768da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5c7b12e4dc3943a299fc8a49cbc02694. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:50.670504 11260 consensus_queue.cc:1035] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.670630 6920 raft_consensus.cc:1261] T 5c7b12e4dc3943a299fc8a49cbc02694 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:50.671600 11260 consensus_queue.cc:1035] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.672233 11471 raft_consensus.cc:2932] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:50.674417 6920 raft_consensus.cc:2932] T 5c7b12e4dc3943a299fc8a49cbc02694 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:50.674887 6790 raft_consensus.cc:2932] T 5c7b12e4dc3943a299fc8a49cbc02694 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:50.679472 6387 catalog_manager.cc:5484] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:50.685276 9205 consensus_queue.cc:237] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
W20240503 13:59:50.687183 9147 consensus_peers.cc:485] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): Couldn't send request to peer 8d574fb4a89d4eea947c859470d44dd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e2ae8c7089db42199a2e6071800d9efc. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:50.687409 6527 raft_consensus.cc:1261] T e2ae8c7089db42199a2e6071800d9efc P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:50.687757 8948 raft_consensus.cc:1261] T e2ae8c7089db42199a2e6071800d9efc P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:50.687844 11260 consensus_queue.cc:1035] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.688341 11260 consensus_queue.cc:1035] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.689615 11665 raft_consensus.cc:2932] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:50.690025 6527 raft_consensus.cc:2932] T e2ae8c7089db42199a2e6071800d9efc P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:50.692854 8948 raft_consensus.cc:2932] T e2ae8c7089db42199a2e6071800d9efc P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:50.694440 6381 catalog_manager.cc:5484] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index -1 to 2, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:50.704527 9343 consensus_queue.cc:237] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:50.705916 6790 raft_consensus.cc:1261] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:50.705922 9205 raft_consensus.cc:1261] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:50.706171 9279 consensus_peers.cc:485] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 -> Peer 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): Couldn't send request to peer 2145d15ab2d74c0287839266f1411a43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d25b0d68b7643cbbbdeb787cac7a1a4. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:50.706338 11378 consensus_queue.cc:1035] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.706501 11690 consensus_queue.cc:1035] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.708129 11378 raft_consensus.cc:2932] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:50.708473 6790 raft_consensus.cc:2932] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:50.708559 9205 raft_consensus.cc:2932] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:50.712600 6381 catalog_manager.cc:5484] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:50.718273 9080 consensus_queue.cc:237] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:50.719694 9205 raft_consensus.cc:1261] T 62c77ec99cf443d2b7d48b12c8855883 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:50.719797 6527 raft_consensus.cc:1261] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:50.720125 11443 consensus_queue.cc:1035] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 13:59:50.720321 9015 consensus_peers.cc:485] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62c77ec99cf443d2b7d48b12c8855883. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:50.720342 11583 consensus_queue.cc:1035] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.721997 11371 raft_consensus.cc:2932] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:50.723718 6527 raft_consensus.cc:2932] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:50.723774 9205 raft_consensus.cc:2932] T 62c77ec99cf443d2b7d48b12c8855883 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:50.728304 6384 catalog_manager.cc:5484] T 62c77ec99cf443d2b7d48b12c8855883 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:50.735841 9343 consensus_queue.cc:237] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:50.737247 6527 raft_consensus.cc:1261] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:50.737417 6920 raft_consensus.cc:1261] T 3d9906a118744dd6a78d50e52f1961e2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:50.737727 11167 consensus_queue.cc:1035] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 13:59:50.738022 9278 consensus_peers.cc:485] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 -> Peer 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): Couldn't send request to peer 8d574fb4a89d4eea947c859470d44dd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d9906a118744dd6a78d50e52f1961e2. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:50.738284 11167 consensus_queue.cc:1035] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.740226 11172 raft_consensus.cc:2932] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:50.740568 6920 raft_consensus.cc:2932] T 3d9906a118744dd6a78d50e52f1961e2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:50.740983 6527 raft_consensus.cc:2932] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:50.744118 6381 catalog_manager.cc:5484] T 3d9906a118744dd6a78d50e52f1961e2 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index -1 to 2, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:50.755703 8948 consensus_queue.cc:237] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:50.757180 9343 raft_consensus.cc:1261] T bccac59f680046389bbe23ff262a82d6 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:50.757517 9080 raft_consensus.cc:1261] T bccac59f680046389bbe23ff262a82d6 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:50.757580 8883 consensus_peers.cc:485] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): Couldn't send request to peer 5f46e19644fd4f4b825c86ba6af768da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bccac59f680046389bbe23ff262a82d6. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:50.757777 11206 consensus_queue.cc:1035] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.758019 11304 consensus_queue.cc:1035] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:50.760282 11206 raft_consensus.cc:2932] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:50.760571 9080 raft_consensus.cc:2932] T bccac59f680046389bbe23ff262a82d6 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:50.763562 6384 catalog_manager.cc:5484] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:50.764487 9343 raft_consensus.cc:2932] T bccac59f680046389bbe23ff262a82d6 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:50.886026 11848 ts_tablet_manager.cc:902] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4: Initiating tablet copy from peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:50.886855 11848 tablet_copy_client.cc:308] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:41931
I20240503 13:59:50.887588 9363 tablet_copy_service.cc:140] P 0d77752181e944698e809e325e5937e6: Received BeginTabletCopySession request for tablet 96ce36eb7c25413eaae380f32b465df7 from peer 8d574fb4a89d4eea947c859470d44dd4 ({username='slave'} at 127.0.99.134:57883)
I20240503 13:59:50.887724 9363 tablet_copy_service.cc:161] P 0d77752181e944698e809e325e5937e6: Beginning new tablet copy session on tablet 96ce36eb7c25413eaae380f32b465df7 from peer 8d574fb4a89d4eea947c859470d44dd4 at {username='slave'} at 127.0.99.134:57883: session id = 8d574fb4a89d4eea947c859470d44dd4-96ce36eb7c25413eaae380f32b465df7
I20240503 13:59:50.889490 9363 tablet_copy_source_session.cc:205] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:50.891825 11848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ce36eb7c25413eaae380f32b465df7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:50.893736 11858 ts_tablet_manager.cc:902] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43: Initiating tablet copy from peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:50.894765 11858 tablet_copy_client.cc:308] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:41931
I20240503 13:59:50.895171 11848 tablet_copy_client.cc:783] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:50.895392 9363 tablet_copy_service.cc:140] P 0d77752181e944698e809e325e5937e6: Received BeginTabletCopySession request for tablet 0fa4727297da40eb9f58f5daa8150e0e from peer 2145d15ab2d74c0287839266f1411a43 ({username='slave'} at 127.0.99.130:49785)
I20240503 13:59:50.895481 9363 tablet_copy_service.cc:161] P 0d77752181e944698e809e325e5937e6: Beginning new tablet copy session on tablet 0fa4727297da40eb9f58f5daa8150e0e from peer 2145d15ab2d74c0287839266f1411a43 at {username='slave'} at 127.0.99.130:49785: session id = 2145d15ab2d74c0287839266f1411a43-0fa4727297da40eb9f58f5daa8150e0e
I20240503 13:59:50.895556 11848 tablet_copy_client.cc:647] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:50.896283 9363 tablet_copy_source_session.cc:205] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:50.897225 11848 tablet_copy_client.cc:523] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:50.898402 11858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fa4727297da40eb9f58f5daa8150e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:50.898742 11848 tablet_bootstrap.cc:492] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:50.900808 11858 tablet_copy_client.cc:783] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:50.900995 11858 tablet_copy_client.cc:647] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:50.902310 11858 tablet_copy_client.cc:523] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:50.903996 11858 tablet_bootstrap.cc:492] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:50.908102 11848 tablet_bootstrap.cc:492] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:50.908155 11858 tablet_bootstrap.cc:492] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:50.908380 11848 tablet_bootstrap.cc:492] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap complete.
I20240503 13:59:50.908411 11858 tablet_bootstrap.cc:492] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43: Bootstrap complete.
I20240503 13:59:50.908540 11848 ts_tablet_manager.cc:1367] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20240503 13:59:50.908571 11858 ts_tablet_manager.cc:1367] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 13:59:50.909003 11848 raft_consensus.cc:348] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:50.908998 11858 raft_consensus.cc:348] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:50.909111 11848 raft_consensus.cc:729] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: LEARNER
I20240503 13:59:50.909112 11858 raft_consensus.cc:729] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: LEARNER
I20240503 13:59:50.909219 11848 consensus_queue.cc:260] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:50.909224 11858 consensus_queue.cc:260] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:50.909494 11858 ts_tablet_manager.cc:1398] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:50.910028 9362 tablet_copy_service.cc:342] P 0d77752181e944698e809e325e5937e6: Request end of tablet copy session 2145d15ab2d74c0287839266f1411a43-0fa4727297da40eb9f58f5daa8150e0e received from {username='slave'} at 127.0.99.130:49785
I20240503 13:59:50.910115 9362 tablet_copy_service.cc:433] P 0d77752181e944698e809e325e5937e6: ending tablet copy session 2145d15ab2d74c0287839266f1411a43-0fa4727297da40eb9f58f5daa8150e0e on tablet 0fa4727297da40eb9f58f5daa8150e0e with peer 2145d15ab2d74c0287839266f1411a43
I20240503 13:59:50.910497 11848 ts_tablet_manager.cc:1398] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:50.912549 9362 tablet_copy_service.cc:342] P 0d77752181e944698e809e325e5937e6: Request end of tablet copy session 8d574fb4a89d4eea947c859470d44dd4-96ce36eb7c25413eaae380f32b465df7 received from {username='slave'} at 127.0.99.134:57883
I20240503 13:59:50.912621 9362 tablet_copy_service.cc:433] P 0d77752181e944698e809e325e5937e6: ending tablet copy session 8d574fb4a89d4eea947c859470d44dd4-96ce36eb7c25413eaae380f32b465df7 on tablet 96ce36eb7c25413eaae380f32b465df7 with peer 8d574fb4a89d4eea947c859470d44dd4
I20240503 13:59:50.920112 11865 ts_tablet_manager.cc:902] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735: Initiating tablet copy from peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:50.920837 11865 tablet_copy_client.cc:308] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43531
I20240503 13:59:50.921649 6942 tablet_copy_service.cc:140] P f58065db70b64ef8b1dc499aa6ae4460: Received BeginTabletCopySession request for tablet 350ba659df804b95b0692091dc2f68a2 from peer ab1d21935d6049888eb0ceccc3c9d735 ({username='slave'} at 127.0.99.135:34893)
I20240503 13:59:50.921792 6942 tablet_copy_service.cc:161] P f58065db70b64ef8b1dc499aa6ae4460: Beginning new tablet copy session on tablet 350ba659df804b95b0692091dc2f68a2 from peer ab1d21935d6049888eb0ceccc3c9d735 at {username='slave'} at 127.0.99.135:34893: session id = ab1d21935d6049888eb0ceccc3c9d735-350ba659df804b95b0692091dc2f68a2
I20240503 13:59:50.922998 6942 tablet_copy_source_session.cc:205] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:50.924459 11865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 350ba659df804b95b0692091dc2f68a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:50.926970 11865 tablet_copy_client.cc:783] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:50.927167 11865 tablet_copy_client.cc:647] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:50.928725 11865 tablet_copy_client.cc:523] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:50.930006 11865 tablet_bootstrap.cc:492] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:50.944363 11871 ts_tablet_manager.cc:902] T 8f57f1ad5c604b5e957374bab029a5df P ab1d21935d6049888eb0ceccc3c9d735: Initiating tablet copy from peer 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:50.944924 11871 tablet_copy_client.cc:308] T 8f57f1ad5c604b5e957374bab029a5df P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46119
I20240503 13:59:50.945622 6680 tablet_copy_service.cc:140] P 2145d15ab2d74c0287839266f1411a43: Received BeginTabletCopySession request for tablet 8f57f1ad5c604b5e957374bab029a5df from peer ab1d21935d6049888eb0ceccc3c9d735 ({username='slave'} at 127.0.99.135:52321)
I20240503 13:59:50.945745 6680 tablet_copy_service.cc:161] P 2145d15ab2d74c0287839266f1411a43: Beginning new tablet copy session on tablet 8f57f1ad5c604b5e957374bab029a5df from peer ab1d21935d6049888eb0ceccc3c9d735 at {username='slave'} at 127.0.99.135:52321: session id = ab1d21935d6049888eb0ceccc3c9d735-8f57f1ad5c604b5e957374bab029a5df
I20240503 13:59:50.947019 6680 tablet_copy_source_session.cc:205] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:50.948233 11871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f57f1ad5c604b5e957374bab029a5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:50.951623 11871 tablet_copy_client.cc:783] T 8f57f1ad5c604b5e957374bab029a5df P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:50.951821 11871 tablet_copy_client.cc:647] T 8f57f1ad5c604b5e957374bab029a5df P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:50.952407 11876 ts_tablet_manager.cc:902] T dfa48440020d4754be0f5ddf159c23fe P 0d77752181e944698e809e325e5937e6: Initiating tablet copy from peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:50.953159 11876 tablet_copy_client.cc:308] T dfa48440020d4754be0f5ddf159c23fe P 0d77752181e944698e809e325e5937e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43531
I20240503 13:59:50.953498 11871 tablet_copy_client.cc:523] T 8f57f1ad5c604b5e957374bab029a5df P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:50.953637 6942 tablet_copy_service.cc:140] P f58065db70b64ef8b1dc499aa6ae4460: Received BeginTabletCopySession request for tablet dfa48440020d4754be0f5ddf159c23fe from peer 0d77752181e944698e809e325e5937e6 ({username='slave'} at 127.0.99.136:51497)
I20240503 13:59:50.953739 6942 tablet_copy_service.cc:161] P f58065db70b64ef8b1dc499aa6ae4460: Beginning new tablet copy session on tablet dfa48440020d4754be0f5ddf159c23fe from peer 0d77752181e944698e809e325e5937e6 at {username='slave'} at 127.0.99.136:51497: session id = 0d77752181e944698e809e325e5937e6-dfa48440020d4754be0f5ddf159c23fe
I20240503 13:59:50.954620 11865 tablet_bootstrap.cc:492] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:50.954648 11871 tablet_bootstrap.cc:492] T 8f57f1ad5c604b5e957374bab029a5df P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:50.954723 6942 tablet_copy_source_session.cc:205] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:50.954928 11865 tablet_bootstrap.cc:492] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap complete.
I20240503 13:59:50.955130 11865 ts_tablet_manager.cc:1367] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.025s user 0.016s sys 0.005s
I20240503 13:59:50.955575 11865 raft_consensus.cc:348] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:50.955677 11865 raft_consensus.cc:729] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: LEARNER
I20240503 13:59:50.955775 11865 consensus_queue.cc:260] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:50.955940 11876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa48440020d4754be0f5ddf159c23fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:50.956152 11865 ts_tablet_manager.cc:1398] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240503 13:59:50.956710 6942 tablet_copy_service.cc:342] P f58065db70b64ef8b1dc499aa6ae4460: Request end of tablet copy session ab1d21935d6049888eb0ceccc3c9d735-350ba659df804b95b0692091dc2f68a2 received from {username='slave'} at 127.0.99.135:34893
I20240503 13:59:50.956794 6942 tablet_copy_service.cc:433] P f58065db70b64ef8b1dc499aa6ae4460: ending tablet copy session ab1d21935d6049888eb0ceccc3c9d735-350ba659df804b95b0692091dc2f68a2 on tablet 350ba659df804b95b0692091dc2f68a2 with peer ab1d21935d6049888eb0ceccc3c9d735
I20240503 13:59:50.958366 11876 tablet_copy_client.cc:783] T dfa48440020d4754be0f5ddf159c23fe P 0d77752181e944698e809e325e5937e6: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:50.958551 11876 tablet_copy_client.cc:647] T dfa48440020d4754be0f5ddf159c23fe P 0d77752181e944698e809e325e5937e6: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:50.960397 11876 tablet_copy_client.cc:523] T dfa48440020d4754be0f5ddf159c23fe P 0d77752181e944698e809e325e5937e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:50.962198 11876 tablet_bootstrap.cc:492] T dfa48440020d4754be0f5ddf159c23fe P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:50.964403 11858 ts_tablet_manager.cc:902] T f853508f20d4431bb35e3e05b659953a P 2145d15ab2d74c0287839266f1411a43: Initiating tablet copy from peer ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:50.964838 11858 tablet_copy_client.cc:308] T f853508f20d4431bb35e3e05b659953a P 2145d15ab2d74c0287839266f1411a43: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:34091
I20240503 13:59:50.965371 9232 tablet_copy_service.cc:140] P ab1d21935d6049888eb0ceccc3c9d735: Received BeginTabletCopySession request for tablet f853508f20d4431bb35e3e05b659953a from peer 2145d15ab2d74c0287839266f1411a43 ({username='slave'} at 127.0.99.130:36297)
I20240503 13:59:50.965490 9232 tablet_copy_service.cc:161] P ab1d21935d6049888eb0ceccc3c9d735: Beginning new tablet copy session on tablet f853508f20d4431bb35e3e05b659953a from peer 2145d15ab2d74c0287839266f1411a43 at {username='slave'} at 127.0.99.130:36297: session id = 2145d15ab2d74c0287839266f1411a43-f853508f20d4431bb35e3e05b659953a
I20240503 13:59:50.966554 9232 tablet_copy_source_session.cc:205] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:50.967525 11858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f853508f20d4431bb35e3e05b659953a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:50.968930 11876 tablet_bootstrap.cc:492] T dfa48440020d4754be0f5ddf159c23fe P 0d77752181e944698e809e325e5937e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:50.969254 11876 tablet_bootstrap.cc:492] T dfa48440020d4754be0f5ddf159c23fe P 0d77752181e944698e809e325e5937e6: Bootstrap complete.
I20240503 13:59:50.969419 11876 ts_tablet_manager.cc:1367] T dfa48440020d4754be0f5ddf159c23fe P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 13:59:50.969738 11858 tablet_copy_client.cc:783] T f853508f20d4431bb35e3e05b659953a P 2145d15ab2d74c0287839266f1411a43: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:50.969892 11858 tablet_copy_client.cc:647] T f853508f20d4431bb35e3e05b659953a P 2145d15ab2d74c0287839266f1411a43: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:50.970007 11876 raft_consensus.cc:348] T dfa48440020d4754be0f5ddf159c23fe P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:50.970233 11876 raft_consensus.cc:729] T dfa48440020d4754be0f5ddf159c23fe P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: LEARNER
I20240503 13:59:50.970360 11876 consensus_queue.cc:260] T dfa48440020d4754be0f5ddf159c23fe P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:50.970654 11876 ts_tablet_manager.cc:1398] T dfa48440020d4754be0f5ddf159c23fe P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:50.971202 11858 tablet_copy_client.cc:523] T f853508f20d4431bb35e3e05b659953a P 2145d15ab2d74c0287839266f1411a43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:50.971455 6942 tablet_copy_service.cc:342] P f58065db70b64ef8b1dc499aa6ae4460: Request end of tablet copy session 0d77752181e944698e809e325e5937e6-dfa48440020d4754be0f5ddf159c23fe received from {username='slave'} at 127.0.99.136:51497
I20240503 13:59:50.971545 6942 tablet_copy_service.cc:433] P f58065db70b64ef8b1dc499aa6ae4460: ending tablet copy session 0d77752181e944698e809e325e5937e6-dfa48440020d4754be0f5ddf159c23fe on tablet dfa48440020d4754be0f5ddf159c23fe with peer 0d77752181e944698e809e325e5937e6
I20240503 13:59:50.972420 11858 tablet_bootstrap.cc:492] T f853508f20d4431bb35e3e05b659953a P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:50.981860 11871 tablet_bootstrap.cc:492] T 8f57f1ad5c604b5e957374bab029a5df P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:50.982132 11871 tablet_bootstrap.cc:492] T 8f57f1ad5c604b5e957374bab029a5df P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap complete.
I20240503 13:59:50.982292 11871 ts_tablet_manager.cc:1367] T 8f57f1ad5c604b5e957374bab029a5df P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.007s
I20240503 13:59:50.982659 11871 raft_consensus.cc:348] T 8f57f1ad5c604b5e957374bab029a5df P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:50.982792 11871 raft_consensus.cc:729] T 8f57f1ad5c604b5e957374bab029a5df P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: LEARNER
I20240503 13:59:50.982895 11871 consensus_queue.cc:260] T 8f57f1ad5c604b5e957374bab029a5df P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:50.983258 11871 ts_tablet_manager.cc:1398] T 8f57f1ad5c604b5e957374bab029a5df P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.003s sys 0.001s
I20240503 13:59:50.983799 6680 tablet_copy_service.cc:342] P 2145d15ab2d74c0287839266f1411a43: Request end of tablet copy session ab1d21935d6049888eb0ceccc3c9d735-8f57f1ad5c604b5e957374bab029a5df received from {username='slave'} at 127.0.99.135:52321
I20240503 13:59:50.983877 6680 tablet_copy_service.cc:433] P 2145d15ab2d74c0287839266f1411a43: ending tablet copy session ab1d21935d6049888eb0ceccc3c9d735-8f57f1ad5c604b5e957374bab029a5df on tablet 8f57f1ad5c604b5e957374bab029a5df with peer ab1d21935d6049888eb0ceccc3c9d735
I20240503 13:59:50.985049 11858 tablet_bootstrap.cc:492] T f853508f20d4431bb35e3e05b659953a P 2145d15ab2d74c0287839266f1411a43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:50.985358 11858 tablet_bootstrap.cc:492] T f853508f20d4431bb35e3e05b659953a P 2145d15ab2d74c0287839266f1411a43: Bootstrap complete.
I20240503 13:59:50.985523 11858 ts_tablet_manager.cc:1367] T f853508f20d4431bb35e3e05b659953a P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240503 13:59:50.985985 11858 raft_consensus.cc:348] T f853508f20d4431bb35e3e05b659953a P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:50.986104 11858 raft_consensus.cc:729] T f853508f20d4431bb35e3e05b659953a P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: LEARNER
I20240503 13:59:50.986202 11858 consensus_queue.cc:260] T f853508f20d4431bb35e3e05b659953a P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:50.986510 11858 ts_tablet_manager.cc:1398] T f853508f20d4431bb35e3e05b659953a P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:50.987380 9232 tablet_copy_service.cc:342] P ab1d21935d6049888eb0ceccc3c9d735: Request end of tablet copy session 2145d15ab2d74c0287839266f1411a43-f853508f20d4431bb35e3e05b659953a received from {username='slave'} at 127.0.99.130:36297
I20240503 13:59:50.987469 9232 tablet_copy_service.cc:433] P ab1d21935d6049888eb0ceccc3c9d735: ending tablet copy session 2145d15ab2d74c0287839266f1411a43-f853508f20d4431bb35e3e05b659953a on tablet f853508f20d4431bb35e3e05b659953a with peer 2145d15ab2d74c0287839266f1411a43
I20240503 13:59:50.991652 11888 ts_tablet_manager.cc:902] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545: Initiating tablet copy from peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:50.992648 11888 tablet_copy_client.cc:308] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:41931
I20240503 13:59:50.993340 9362 tablet_copy_service.cc:140] P 0d77752181e944698e809e325e5937e6: Received BeginTabletCopySession request for tablet 0fe53d65ab4d482aad118d6e3bd5533c from peer 06ca8dbc004c42b99faa527a4e6f9545 ({username='slave'} at 127.0.99.133:45179)
I20240503 13:59:50.993436 9362 tablet_copy_service.cc:161] P 0d77752181e944698e809e325e5937e6: Beginning new tablet copy session on tablet 0fe53d65ab4d482aad118d6e3bd5533c from peer 06ca8dbc004c42b99faa527a4e6f9545 at {username='slave'} at 127.0.99.133:45179: session id = 06ca8dbc004c42b99faa527a4e6f9545-0fe53d65ab4d482aad118d6e3bd5533c
I20240503 13:59:50.994248 9362 tablet_copy_source_session.cc:205] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:50.995378 11888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fe53d65ab4d482aad118d6e3bd5533c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:50.997407 11888 tablet_copy_client.cc:783] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:50.997613 11888 tablet_copy_client.cc:647] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:50.998916 11888 tablet_copy_client.cc:523] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:51.000691 11888 tablet_bootstrap.cc:492] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:51.005228 11888 tablet_bootstrap.cc:492] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:51.005496 11888 tablet_bootstrap.cc:492] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap complete.
I20240503 13:59:51.005648 11888 ts_tablet_manager.cc:1367] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240503 13:59:51.006687 11888 raft_consensus.cc:348] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 13:59:51.006812 11888 raft_consensus.cc:729] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: LEARNER
I20240503 13:59:51.006896 11888 consensus_queue.cc:260] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 13:59:51.007161 11888 ts_tablet_manager.cc:1398] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 13:59:51.007689 9362 tablet_copy_service.cc:342] P 0d77752181e944698e809e325e5937e6: Request end of tablet copy session 06ca8dbc004c42b99faa527a4e6f9545-0fe53d65ab4d482aad118d6e3bd5533c received from {username='slave'} at 127.0.99.133:45179
I20240503 13:59:51.007782 9362 tablet_copy_service.cc:433] P 0d77752181e944698e809e325e5937e6: ending tablet copy session 06ca8dbc004c42b99faa527a4e6f9545-0fe53d65ab4d482aad118d6e3bd5533c on tablet 0fe53d65ab4d482aad118d6e3bd5533c with peer 06ca8dbc004c42b99faa527a4e6f9545
I20240503 13:59:51.034662 11895 ts_tablet_manager.cc:902] T a7e9c80dbf8c41469e54535c20648df4 P 4a85e492c8154e9881a9cc2a105d7e64: Initiating tablet copy from peer ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:51.035468 11895 tablet_copy_client.cc:308] T a7e9c80dbf8c41469e54535c20648df4 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:34091
I20240503 13:59:51.036063 9232 tablet_copy_service.cc:140] P ab1d21935d6049888eb0ceccc3c9d735: Received BeginTabletCopySession request for tablet a7e9c80dbf8c41469e54535c20648df4 from peer 4a85e492c8154e9881a9cc2a105d7e64 ({username='slave'} at 127.0.99.131:60171)
I20240503 13:59:51.036164 9232 tablet_copy_service.cc:161] P ab1d21935d6049888eb0ceccc3c9d735: Beginning new tablet copy session on tablet a7e9c80dbf8c41469e54535c20648df4 from peer 4a85e492c8154e9881a9cc2a105d7e64 at {username='slave'} at 127.0.99.131:60171: session id = 4a85e492c8154e9881a9cc2a105d7e64-a7e9c80dbf8c41469e54535c20648df4
I20240503 13:59:51.036902 9232 tablet_copy_source_session.cc:205] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:51.037504 11858 ts_tablet_manager.cc:902] T 1eec99c0bb994dd7b6ae63918125bccb P 2145d15ab2d74c0287839266f1411a43: Initiating tablet copy from peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:51.037961 11858 tablet_copy_client.cc:308] T 1eec99c0bb994dd7b6ae63918125bccb P 2145d15ab2d74c0287839266f1411a43: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:41931
I20240503 13:59:51.038434 11895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7e9c80dbf8c41469e54535c20648df4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:51.041275 9362 tablet_copy_service.cc:140] P 0d77752181e944698e809e325e5937e6: Received BeginTabletCopySession request for tablet 1eec99c0bb994dd7b6ae63918125bccb from peer 2145d15ab2d74c0287839266f1411a43 ({username='slave'} at 127.0.99.130:49785)
I20240503 13:59:51.041393 9362 tablet_copy_service.cc:161] P 0d77752181e944698e809e325e5937e6: Beginning new tablet copy session on tablet 1eec99c0bb994dd7b6ae63918125bccb from peer 2145d15ab2d74c0287839266f1411a43 at {username='slave'} at 127.0.99.130:49785: session id = 2145d15ab2d74c0287839266f1411a43-1eec99c0bb994dd7b6ae63918125bccb
I20240503 13:59:51.041707 11895 tablet_copy_client.cc:783] T a7e9c80dbf8c41469e54535c20648df4 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:51.041913 11895 tablet_copy_client.cc:647] T a7e9c80dbf8c41469e54535c20648df4 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:51.042353 9362 tablet_copy_source_session.cc:205] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:51.042465 11904 ts_tablet_manager.cc:902] T 1350225b1a2848debca68e1bad5528c5 P 4a85e492c8154e9881a9cc2a105d7e64: Initiating tablet copy from peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:51.042910 11904 tablet_copy_client.cc:308] T 1350225b1a2848debca68e1bad5528c5 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:41931
I20240503 13:59:51.043462 9362 tablet_copy_service.cc:140] P 0d77752181e944698e809e325e5937e6: Received BeginTabletCopySession request for tablet 1350225b1a2848debca68e1bad5528c5 from peer 4a85e492c8154e9881a9cc2a105d7e64 ({username='slave'} at 127.0.99.131:49089)
I20240503 13:59:51.043510 11858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eec99c0bb994dd7b6ae63918125bccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:51.043547 9362 tablet_copy_service.cc:161] P 0d77752181e944698e809e325e5937e6: Beginning new tablet copy session on tablet 1350225b1a2848debca68e1bad5528c5 from peer 4a85e492c8154e9881a9cc2a105d7e64 at {username='slave'} at 127.0.99.131:49089: session id = 4a85e492c8154e9881a9cc2a105d7e64-1350225b1a2848debca68e1bad5528c5
I20240503 13:59:51.044261 9362 tablet_copy_source_session.cc:205] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:51.045753 11858 tablet_copy_client.cc:783] T 1eec99c0bb994dd7b6ae63918125bccb P 2145d15ab2d74c0287839266f1411a43: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:51.045910 11858 tablet_copy_client.cc:647] T 1eec99c0bb994dd7b6ae63918125bccb P 2145d15ab2d74c0287839266f1411a43: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:51.046036 11895 tablet_copy_client.cc:523] T a7e9c80dbf8c41469e54535c20648df4 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:51.046051 11904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1350225b1a2848debca68e1bad5528c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:51.047397 11858 tablet_copy_client.cc:523] T 1eec99c0bb994dd7b6ae63918125bccb P 2145d15ab2d74c0287839266f1411a43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:51.047609 11895 tablet_bootstrap.cc:492] T a7e9c80dbf8c41469e54535c20648df4 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:51.048147 11904 tablet_copy_client.cc:783] T 1350225b1a2848debca68e1bad5528c5 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:51.048301 11904 tablet_copy_client.cc:647] T 1350225b1a2848debca68e1bad5528c5 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:51.048636 11858 tablet_bootstrap.cc:492] T 1eec99c0bb994dd7b6ae63918125bccb P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:51.051523 11904 tablet_copy_client.cc:523] T 1350225b1a2848debca68e1bad5528c5 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:51.052500 11858 tablet_bootstrap.cc:492] T 1eec99c0bb994dd7b6ae63918125bccb P 2145d15ab2d74c0287839266f1411a43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:51.052762 11858 tablet_bootstrap.cc:492] T 1eec99c0bb994dd7b6ae63918125bccb P 2145d15ab2d74c0287839266f1411a43: Bootstrap complete.
I20240503 13:59:51.052949 11858 ts_tablet_manager.cc:1367] T 1eec99c0bb994dd7b6ae63918125bccb P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240503 13:59:51.053241 11904 tablet_bootstrap.cc:492] T 1350225b1a2848debca68e1bad5528c5 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:51.053458 11858 raft_consensus.cc:348] T 1eec99c0bb994dd7b6ae63918125bccb P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:51.053633 11858 raft_consensus.cc:729] T 1eec99c0bb994dd7b6ae63918125bccb P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: LEARNER
I20240503 13:59:51.053742 11858 consensus_queue.cc:260] T 1eec99c0bb994dd7b6ae63918125bccb P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:51.053911 11895 tablet_bootstrap.cc:492] T a7e9c80dbf8c41469e54535c20648df4 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:51.054157 11858 ts_tablet_manager.cc:1398] T 1eec99c0bb994dd7b6ae63918125bccb P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:51.054198 11895 tablet_bootstrap.cc:492] T a7e9c80dbf8c41469e54535c20648df4 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap complete.
I20240503 13:59:51.054381 11895 ts_tablet_manager.cc:1367] T a7e9c80dbf8c41469e54535c20648df4 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20240503 13:59:51.054883 9362 tablet_copy_service.cc:342] P 0d77752181e944698e809e325e5937e6: Request end of tablet copy session 2145d15ab2d74c0287839266f1411a43-1eec99c0bb994dd7b6ae63918125bccb received from {username='slave'} at 127.0.99.130:49785
I20240503 13:59:51.054975 9362 tablet_copy_service.cc:433] P 0d77752181e944698e809e325e5937e6: ending tablet copy session 2145d15ab2d74c0287839266f1411a43-1eec99c0bb994dd7b6ae63918125bccb on tablet 1eec99c0bb994dd7b6ae63918125bccb with peer 2145d15ab2d74c0287839266f1411a43
I20240503 13:59:51.055630 11895 raft_consensus.cc:348] T a7e9c80dbf8c41469e54535c20648df4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:51.055778 11895 raft_consensus.cc:729] T a7e9c80dbf8c41469e54535c20648df4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: LEARNER
I20240503 13:59:51.055914 11895 consensus_queue.cc:260] T a7e9c80dbf8c41469e54535c20648df4 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:51.056164 11895 ts_tablet_manager.cc:1398] T a7e9c80dbf8c41469e54535c20648df4 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:51.056373 11911 ts_tablet_manager.cc:902] T 7a44f5349cc44be69e9b52f8047d2bb1 P 4a85e492c8154e9881a9cc2a105d7e64: Initiating tablet copy from peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:51.056855 11911 tablet_copy_client.cc:308] T 7a44f5349cc44be69e9b52f8047d2bb1 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43531
I20240503 13:59:51.057463 6942 tablet_copy_service.cc:140] P f58065db70b64ef8b1dc499aa6ae4460: Received BeginTabletCopySession request for tablet 7a44f5349cc44be69e9b52f8047d2bb1 from peer 4a85e492c8154e9881a9cc2a105d7e64 ({username='slave'} at 127.0.99.131:38291)
I20240503 13:59:51.057571 6942 tablet_copy_service.cc:161] P f58065db70b64ef8b1dc499aa6ae4460: Beginning new tablet copy session on tablet 7a44f5349cc44be69e9b52f8047d2bb1 from peer 4a85e492c8154e9881a9cc2a105d7e64 at {username='slave'} at 127.0.99.131:38291: session id = 4a85e492c8154e9881a9cc2a105d7e64-7a44f5349cc44be69e9b52f8047d2bb1
I20240503 13:59:51.057652 9232 tablet_copy_service.cc:342] P ab1d21935d6049888eb0ceccc3c9d735: Request end of tablet copy session 4a85e492c8154e9881a9cc2a105d7e64-a7e9c80dbf8c41469e54535c20648df4 received from {username='slave'} at 127.0.99.131:60171
I20240503 13:59:51.057727 9232 tablet_copy_service.cc:433] P ab1d21935d6049888eb0ceccc3c9d735: ending tablet copy session 4a85e492c8154e9881a9cc2a105d7e64-a7e9c80dbf8c41469e54535c20648df4 on tablet a7e9c80dbf8c41469e54535c20648df4 with peer 4a85e492c8154e9881a9cc2a105d7e64
I20240503 13:59:51.058578 6942 tablet_copy_source_session.cc:205] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:51.059883 11911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a44f5349cc44be69e9b52f8047d2bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:51.059935 11904 tablet_bootstrap.cc:492] T 1350225b1a2848debca68e1bad5528c5 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:51.060236 11904 tablet_bootstrap.cc:492] T 1350225b1a2848debca68e1bad5528c5 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap complete.
I20240503 13:59:51.060458 11904 ts_tablet_manager.cc:1367] T 1350225b1a2848debca68e1bad5528c5 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240503 13:59:51.060943 11904 raft_consensus.cc:348] T 1350225b1a2848debca68e1bad5528c5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:51.061151 11904 raft_consensus.cc:729] T 1350225b1a2848debca68e1bad5528c5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: LEARNER
I20240503 13:59:51.061403 11904 consensus_queue.cc:260] T 1350225b1a2848debca68e1bad5528c5 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:51.062490 11911 tablet_copy_client.cc:783] T 7a44f5349cc44be69e9b52f8047d2bb1 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:51.062575 11904 ts_tablet_manager.cc:1398] T 1350225b1a2848debca68e1bad5528c5 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:51.062675 11911 tablet_copy_client.cc:647] T 7a44f5349cc44be69e9b52f8047d2bb1 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:51.063386 9362 tablet_copy_service.cc:342] P 0d77752181e944698e809e325e5937e6: Request end of tablet copy session 4a85e492c8154e9881a9cc2a105d7e64-1350225b1a2848debca68e1bad5528c5 received from {username='slave'} at 127.0.99.131:49089
I20240503 13:59:51.063468 9362 tablet_copy_service.cc:433] P 0d77752181e944698e809e325e5937e6: ending tablet copy session 4a85e492c8154e9881a9cc2a105d7e64-1350225b1a2848debca68e1bad5528c5 on tablet 1350225b1a2848debca68e1bad5528c5 with peer 4a85e492c8154e9881a9cc2a105d7e64
I20240503 13:59:51.065362 11911 tablet_copy_client.cc:523] T 7a44f5349cc44be69e9b52f8047d2bb1 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:51.066424 11911 tablet_bootstrap.cc:492] T 7a44f5349cc44be69e9b52f8047d2bb1 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:51.069181 11911 tablet_bootstrap.cc:492] T 7a44f5349cc44be69e9b52f8047d2bb1 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:51.069411 11911 tablet_bootstrap.cc:492] T 7a44f5349cc44be69e9b52f8047d2bb1 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap complete.
I20240503 13:59:51.069567 11911 ts_tablet_manager.cc:1367] T 7a44f5349cc44be69e9b52f8047d2bb1 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240503 13:59:51.069921 11911 raft_consensus.cc:348] T 7a44f5349cc44be69e9b52f8047d2bb1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:51.070012 11911 raft_consensus.cc:729] T 7a44f5349cc44be69e9b52f8047d2bb1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: LEARNER
I20240503 13:59:51.070102 11911 consensus_queue.cc:260] T 7a44f5349cc44be69e9b52f8047d2bb1 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:51.070322 11911 ts_tablet_manager.cc:1398] T 7a44f5349cc44be69e9b52f8047d2bb1 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:51.070757 6942 tablet_copy_service.cc:342] P f58065db70b64ef8b1dc499aa6ae4460: Request end of tablet copy session 4a85e492c8154e9881a9cc2a105d7e64-7a44f5349cc44be69e9b52f8047d2bb1 received from {username='slave'} at 127.0.99.131:38291
I20240503 13:59:51.070844 6942 tablet_copy_service.cc:433] P f58065db70b64ef8b1dc499aa6ae4460: ending tablet copy session 4a85e492c8154e9881a9cc2a105d7e64-7a44f5349cc44be69e9b52f8047d2bb1 on tablet 7a44f5349cc44be69e9b52f8047d2bb1 with peer 4a85e492c8154e9881a9cc2a105d7e64
I20240503 13:59:51.086300 11918 ts_tablet_manager.cc:902] T 5c7b12e4dc3943a299fc8a49cbc02694 P 5f46e19644fd4f4b825c86ba6af768da: Initiating tablet copy from peer ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:51.087196 11918 tablet_copy_client.cc:308] T 5c7b12e4dc3943a299fc8a49cbc02694 P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:34091
I20240503 13:59:51.087785 9232 tablet_copy_service.cc:140] P ab1d21935d6049888eb0ceccc3c9d735: Received BeginTabletCopySession request for tablet 5c7b12e4dc3943a299fc8a49cbc02694 from peer 5f46e19644fd4f4b825c86ba6af768da ({username='slave'} at 127.0.99.129:36631)
I20240503 13:59:51.087924 9232 tablet_copy_service.cc:161] P ab1d21935d6049888eb0ceccc3c9d735: Beginning new tablet copy session on tablet 5c7b12e4dc3943a299fc8a49cbc02694 from peer 5f46e19644fd4f4b825c86ba6af768da at {username='slave'} at 127.0.99.129:36631: session id = 5f46e19644fd4f4b825c86ba6af768da-5c7b12e4dc3943a299fc8a49cbc02694
I20240503 13:59:51.088802 9232 tablet_copy_source_session.cc:205] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:51.090045 11918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c7b12e4dc3943a299fc8a49cbc02694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:51.092180 11918 tablet_copy_client.cc:783] T 5c7b12e4dc3943a299fc8a49cbc02694 P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:51.092360 11918 tablet_copy_client.cc:647] T 5c7b12e4dc3943a299fc8a49cbc02694 P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:51.093706 11918 tablet_copy_client.cc:523] T 5c7b12e4dc3943a299fc8a49cbc02694 P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:51.095137 11918 tablet_bootstrap.cc:492] T 5c7b12e4dc3943a299fc8a49cbc02694 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:51.101903 11918 tablet_bootstrap.cc:492] T 5c7b12e4dc3943a299fc8a49cbc02694 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:51.102223 11918 tablet_bootstrap.cc:492] T 5c7b12e4dc3943a299fc8a49cbc02694 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap complete.
I20240503 13:59:51.102404 11918 ts_tablet_manager.cc:1367] T 5c7b12e4dc3943a299fc8a49cbc02694 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 13:59:51.102895 11918 raft_consensus.cc:348] T 5c7b12e4dc3943a299fc8a49cbc02694 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:51.103044 11918 raft_consensus.cc:729] T 5c7b12e4dc3943a299fc8a49cbc02694 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: LEARNER
I20240503 13:59:51.103166 11918 consensus_queue.cc:260] T 5c7b12e4dc3943a299fc8a49cbc02694 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:51.103475 11918 ts_tablet_manager.cc:1398] T 5c7b12e4dc3943a299fc8a49cbc02694 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.003s
I20240503 13:59:51.104035 9232 tablet_copy_service.cc:342] P ab1d21935d6049888eb0ceccc3c9d735: Request end of tablet copy session 5f46e19644fd4f4b825c86ba6af768da-5c7b12e4dc3943a299fc8a49cbc02694 received from {username='slave'} at 127.0.99.129:36631
I20240503 13:59:51.104115 9232 tablet_copy_service.cc:433] P ab1d21935d6049888eb0ceccc3c9d735: ending tablet copy session 5f46e19644fd4f4b825c86ba6af768da-5c7b12e4dc3943a299fc8a49cbc02694 on tablet 5c7b12e4dc3943a299fc8a49cbc02694 with peer 5f46e19644fd4f4b825c86ba6af768da
I20240503 13:59:51.121806 11918 ts_tablet_manager.cc:902] T 8f9149a9faa746cd9542aab02a8e2850 P 5f46e19644fd4f4b825c86ba6af768da: Initiating tablet copy from peer 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:51.122285 11918 tablet_copy_client.cc:308] T 8f9149a9faa746cd9542aab02a8e2850 P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46119
I20240503 13:59:51.123443 6680 tablet_copy_service.cc:140] P 2145d15ab2d74c0287839266f1411a43: Received BeginTabletCopySession request for tablet 8f9149a9faa746cd9542aab02a8e2850 from peer 5f46e19644fd4f4b825c86ba6af768da ({username='slave'} at 127.0.99.129:60979)
I20240503 13:59:51.123550 6680 tablet_copy_service.cc:161] P 2145d15ab2d74c0287839266f1411a43: Beginning new tablet copy session on tablet 8f9149a9faa746cd9542aab02a8e2850 from peer 5f46e19644fd4f4b825c86ba6af768da at {username='slave'} at 127.0.99.129:60979: session id = 5f46e19644fd4f4b825c86ba6af768da-8f9149a9faa746cd9542aab02a8e2850
I20240503 13:59:51.124490 6680 tablet_copy_source_session.cc:205] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:51.125250 11858 ts_tablet_manager.cc:902] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43: Initiating tablet copy from peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:51.125487 11918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f9149a9faa746cd9542aab02a8e2850. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:51.126646 11858 tablet_copy_client.cc:308] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:41931
I20240503 13:59:51.127120 9362 tablet_copy_service.cc:140] P 0d77752181e944698e809e325e5937e6: Received BeginTabletCopySession request for tablet 8d25b0d68b7643cbbbdeb787cac7a1a4 from peer 2145d15ab2d74c0287839266f1411a43 ({username='slave'} at 127.0.99.130:49785)
I20240503 13:59:51.127225 9362 tablet_copy_service.cc:161] P 0d77752181e944698e809e325e5937e6: Beginning new tablet copy session on tablet 8d25b0d68b7643cbbbdeb787cac7a1a4 from peer 2145d15ab2d74c0287839266f1411a43 at {username='slave'} at 127.0.99.130:49785: session id = 2145d15ab2d74c0287839266f1411a43-8d25b0d68b7643cbbbdeb787cac7a1a4
I20240503 13:59:51.128182 9362 tablet_copy_source_session.cc:205] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:51.128918 11918 tablet_copy_client.cc:783] T 8f9149a9faa746cd9542aab02a8e2850 P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:51.129074 11918 tablet_copy_client.cc:647] T 8f9149a9faa746cd9542aab02a8e2850 P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:51.129484 11858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d25b0d68b7643cbbbdeb787cac7a1a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:51.131577 11858 tablet_copy_client.cc:783] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:51.131705 11858 tablet_copy_client.cc:647] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:51.133296 11858 tablet_copy_client.cc:523] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:51.133438 11918 tablet_copy_client.cc:523] T 8f9149a9faa746cd9542aab02a8e2850 P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:51.134531 11858 tablet_bootstrap.cc:492] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:51.134574 11918 tablet_bootstrap.cc:492] T 8f9149a9faa746cd9542aab02a8e2850 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:51.138553 11918 tablet_bootstrap.cc:492] T 8f9149a9faa746cd9542aab02a8e2850 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:51.138794 11918 tablet_bootstrap.cc:492] T 8f9149a9faa746cd9542aab02a8e2850 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap complete.
I20240503 13:59:51.138944 11918 ts_tablet_manager.cc:1367] T 8f9149a9faa746cd9542aab02a8e2850 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 13:59:51.139382 11918 raft_consensus.cc:348] T 8f9149a9faa746cd9542aab02a8e2850 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:51.139514 11918 raft_consensus.cc:729] T 8f9149a9faa746cd9542aab02a8e2850 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: LEARNER
I20240503 13:59:51.139607 11918 consensus_queue.cc:260] T 8f9149a9faa746cd9542aab02a8e2850 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:51.139868 11918 ts_tablet_manager.cc:1398] T 8f9149a9faa746cd9542aab02a8e2850 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:51.140370 6680 tablet_copy_service.cc:342] P 2145d15ab2d74c0287839266f1411a43: Request end of tablet copy session 5f46e19644fd4f4b825c86ba6af768da-8f9149a9faa746cd9542aab02a8e2850 received from {username='slave'} at 127.0.99.129:60979
I20240503 13:59:51.140455 6680 tablet_copy_service.cc:433] P 2145d15ab2d74c0287839266f1411a43: ending tablet copy session 5f46e19644fd4f4b825c86ba6af768da-8f9149a9faa746cd9542aab02a8e2850 on tablet 8f9149a9faa746cd9542aab02a8e2850 with peer 5f46e19644fd4f4b825c86ba6af768da
I20240503 13:59:51.147964 11964 ts_tablet_manager.cc:902] T b7e1060596204c60882388e984986ac6 P 2145d15ab2d74c0287839266f1411a43: Initiating tablet copy from peer ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:51.148576 11964 tablet_copy_client.cc:308] T b7e1060596204c60882388e984986ac6 P 2145d15ab2d74c0287839266f1411a43: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:34091
I20240503 13:59:51.149336 11858 tablet_bootstrap.cc:492] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:51.149680 11858 tablet_bootstrap.cc:492] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43: Bootstrap complete.
I20240503 13:59:51.149848 11858 ts_tablet_manager.cc:1367] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.002s
I20240503 13:59:51.149989 9232 tablet_copy_service.cc:140] P ab1d21935d6049888eb0ceccc3c9d735: Received BeginTabletCopySession request for tablet b7e1060596204c60882388e984986ac6 from peer 2145d15ab2d74c0287839266f1411a43 ({username='slave'} at 127.0.99.130:36297)
I20240503 13:59:51.150115 9232 tablet_copy_service.cc:161] P ab1d21935d6049888eb0ceccc3c9d735: Beginning new tablet copy session on tablet b7e1060596204c60882388e984986ac6 from peer 2145d15ab2d74c0287839266f1411a43 at {username='slave'} at 127.0.99.130:36297: session id = 2145d15ab2d74c0287839266f1411a43-b7e1060596204c60882388e984986ac6
I20240503 13:59:51.150301 11858 raft_consensus.cc:348] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:51.150414 11858 raft_consensus.cc:729] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: LEARNER
I20240503 13:59:51.150501 11858 consensus_queue.cc:260] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:51.150760 11858 ts_tablet_manager.cc:1398] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:51.150868 9232 tablet_copy_source_session.cc:205] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:51.151962 9362 tablet_copy_service.cc:342] P 0d77752181e944698e809e325e5937e6: Request end of tablet copy session 2145d15ab2d74c0287839266f1411a43-8d25b0d68b7643cbbbdeb787cac7a1a4 received from {username='slave'} at 127.0.99.130:49785
I20240503 13:59:51.152036 9362 tablet_copy_service.cc:433] P 0d77752181e944698e809e325e5937e6: ending tablet copy session 2145d15ab2d74c0287839266f1411a43-8d25b0d68b7643cbbbdeb787cac7a1a4 on tablet 8d25b0d68b7643cbbbdeb787cac7a1a4 with peer 2145d15ab2d74c0287839266f1411a43
I20240503 13:59:51.152271 11848 ts_tablet_manager.cc:902] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4: Initiating tablet copy from peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:51.152743 11848 tablet_copy_client.cc:308] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:41931
I20240503 13:59:51.155526 9362 tablet_copy_service.cc:140] P 0d77752181e944698e809e325e5937e6: Received BeginTabletCopySession request for tablet 3d9906a118744dd6a78d50e52f1961e2 from peer 8d574fb4a89d4eea947c859470d44dd4 ({username='slave'} at 127.0.99.134:57883)
I20240503 13:59:51.155614 9362 tablet_copy_service.cc:161] P 0d77752181e944698e809e325e5937e6: Beginning new tablet copy session on tablet 3d9906a118744dd6a78d50e52f1961e2 from peer 8d574fb4a89d4eea947c859470d44dd4 at {username='slave'} at 127.0.99.134:57883: session id = 8d574fb4a89d4eea947c859470d44dd4-3d9906a118744dd6a78d50e52f1961e2
I20240503 13:59:51.156502 9362 tablet_copy_source_session.cc:205] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:51.158581 11848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d9906a118744dd6a78d50e52f1961e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:51.158946 11973 ts_tablet_manager.cc:902] T e2ae8c7089db42199a2e6071800d9efc P 8d574fb4a89d4eea947c859470d44dd4: Initiating tablet copy from peer ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:51.159505 11973 tablet_copy_client.cc:308] T e2ae8c7089db42199a2e6071800d9efc P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:34091
I20240503 13:59:51.159950 9232 tablet_copy_service.cc:140] P ab1d21935d6049888eb0ceccc3c9d735: Received BeginTabletCopySession request for tablet e2ae8c7089db42199a2e6071800d9efc from peer 8d574fb4a89d4eea947c859470d44dd4 ({username='slave'} at 127.0.99.134:42211)
I20240503 13:59:51.160048 9232 tablet_copy_service.cc:161] P ab1d21935d6049888eb0ceccc3c9d735: Beginning new tablet copy session on tablet e2ae8c7089db42199a2e6071800d9efc from peer 8d574fb4a89d4eea947c859470d44dd4 at {username='slave'} at 127.0.99.134:42211: session id = 8d574fb4a89d4eea947c859470d44dd4-e2ae8c7089db42199a2e6071800d9efc
I20240503 13:59:51.160684 9232 tablet_copy_source_session.cc:205] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:51.160789 11848 tablet_copy_client.cc:783] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:51.160951 11848 tablet_copy_client.cc:647] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:51.161439 11973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2ae8c7089db42199a2e6071800d9efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:51.161873 11964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7e1060596204c60882388e984986ac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:51.163542 11973 tablet_copy_client.cc:783] T e2ae8c7089db42199a2e6071800d9efc P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:51.163728 11973 tablet_copy_client.cc:647] T e2ae8c7089db42199a2e6071800d9efc P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:51.164229 11964 tablet_copy_client.cc:783] T b7e1060596204c60882388e984986ac6 P 2145d15ab2d74c0287839266f1411a43: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:51.164417 11964 tablet_copy_client.cc:647] T b7e1060596204c60882388e984986ac6 P 2145d15ab2d74c0287839266f1411a43: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:51.165786 11848 tablet_copy_client.cc:523] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:51.166028 11973 tablet_copy_client.cc:523] T e2ae8c7089db42199a2e6071800d9efc P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:51.167039 11848 tablet_bootstrap.cc:492] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:51.167310 11973 tablet_bootstrap.cc:492] T e2ae8c7089db42199a2e6071800d9efc P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:51.170173 11964 tablet_copy_client.cc:523] T b7e1060596204c60882388e984986ac6 P 2145d15ab2d74c0287839266f1411a43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:51.171483 11848 tablet_bootstrap.cc:492] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:51.171654 11964 tablet_bootstrap.cc:492] T b7e1060596204c60882388e984986ac6 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:51.171741 11848 tablet_bootstrap.cc:492] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap complete.
I20240503 13:59:51.171918 11848 ts_tablet_manager.cc:1367] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240503 13:59:51.172400 11848 raft_consensus.cc:348] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:51.172534 11848 raft_consensus.cc:729] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: LEARNER
I20240503 13:59:51.172636 11848 consensus_queue.cc:260] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:51.172953 11848 ts_tablet_manager.cc:1398] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:51.173530 11973 tablet_bootstrap.cc:492] T e2ae8c7089db42199a2e6071800d9efc P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:51.173738 11973 tablet_bootstrap.cc:492] T e2ae8c7089db42199a2e6071800d9efc P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap complete.
I20240503 13:59:51.173888 9362 tablet_copy_service.cc:342] P 0d77752181e944698e809e325e5937e6: Request end of tablet copy session 8d574fb4a89d4eea947c859470d44dd4-3d9906a118744dd6a78d50e52f1961e2 received from {username='slave'} at 127.0.99.134:57883
I20240503 13:59:51.173985 9362 tablet_copy_service.cc:433] P 0d77752181e944698e809e325e5937e6: ending tablet copy session 8d574fb4a89d4eea947c859470d44dd4-3d9906a118744dd6a78d50e52f1961e2 on tablet 3d9906a118744dd6a78d50e52f1961e2 with peer 8d574fb4a89d4eea947c859470d44dd4
I20240503 13:59:51.174173 11973 ts_tablet_manager.cc:1367] T e2ae8c7089db42199a2e6071800d9efc P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240503 13:59:51.174602 11973 raft_consensus.cc:348] T e2ae8c7089db42199a2e6071800d9efc P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:51.174686 11973 raft_consensus.cc:729] T e2ae8c7089db42199a2e6071800d9efc P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: LEARNER
I20240503 13:59:51.174780 11973 consensus_queue.cc:260] T e2ae8c7089db42199a2e6071800d9efc P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:51.175118 11973 ts_tablet_manager.cc:1398] T e2ae8c7089db42199a2e6071800d9efc P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:51.175586 9231 tablet_copy_service.cc:342] P ab1d21935d6049888eb0ceccc3c9d735: Request end of tablet copy session 8d574fb4a89d4eea947c859470d44dd4-e2ae8c7089db42199a2e6071800d9efc received from {username='slave'} at 127.0.99.134:42211
I20240503 13:59:51.175663 9231 tablet_copy_service.cc:433] P ab1d21935d6049888eb0ceccc3c9d735: ending tablet copy session 8d574fb4a89d4eea947c859470d44dd4-e2ae8c7089db42199a2e6071800d9efc on tablet e2ae8c7089db42199a2e6071800d9efc with peer 8d574fb4a89d4eea947c859470d44dd4
I20240503 13:59:51.181129 11964 tablet_bootstrap.cc:492] T b7e1060596204c60882388e984986ac6 P 2145d15ab2d74c0287839266f1411a43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:51.181357 11964 tablet_bootstrap.cc:492] T b7e1060596204c60882388e984986ac6 P 2145d15ab2d74c0287839266f1411a43: Bootstrap complete.
I20240503 13:59:51.181506 11964 ts_tablet_manager.cc:1367] T b7e1060596204c60882388e984986ac6 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20240503 13:59:51.181895 11964 raft_consensus.cc:348] T b7e1060596204c60882388e984986ac6 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:51.182015 11964 raft_consensus.cc:729] T b7e1060596204c60882388e984986ac6 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: LEARNER
I20240503 13:59:51.182142 11964 consensus_queue.cc:260] T b7e1060596204c60882388e984986ac6 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:51.182426 11964 ts_tablet_manager.cc:1398] T b7e1060596204c60882388e984986ac6 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 13:59:51.182979 9231 tablet_copy_service.cc:342] P ab1d21935d6049888eb0ceccc3c9d735: Request end of tablet copy session 2145d15ab2d74c0287839266f1411a43-b7e1060596204c60882388e984986ac6 received from {username='slave'} at 127.0.99.130:36297
I20240503 13:59:51.183053 9231 tablet_copy_service.cc:433] P ab1d21935d6049888eb0ceccc3c9d735: ending tablet copy session 2145d15ab2d74c0287839266f1411a43-b7e1060596204c60882388e984986ac6 on tablet b7e1060596204c60882388e984986ac6 with peer 2145d15ab2d74c0287839266f1411a43
I20240503 13:59:51.188012 12001 ts_tablet_manager.cc:902] T ee688e32b7d049509a39259d13439cf6 P f58065db70b64ef8b1dc499aa6ae4460: Initiating tablet copy from peer 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:51.188807 12001 tablet_copy_client.cc:308] T ee688e32b7d049509a39259d13439cf6 P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:39481
I20240503 13:59:51.189754 9101 tablet_copy_service.cc:140] P 8d574fb4a89d4eea947c859470d44dd4: Received BeginTabletCopySession request for tablet ee688e32b7d049509a39259d13439cf6 from peer f58065db70b64ef8b1dc499aa6ae4460 ({username='slave'} at 127.0.99.132:53261)
I20240503 13:59:51.189877 9101 tablet_copy_service.cc:161] P 8d574fb4a89d4eea947c859470d44dd4: Beginning new tablet copy session on tablet ee688e32b7d049509a39259d13439cf6 from peer f58065db70b64ef8b1dc499aa6ae4460 at {username='slave'} at 127.0.99.132:53261: session id = f58065db70b64ef8b1dc499aa6ae4460-ee688e32b7d049509a39259d13439cf6
I20240503 13:59:51.191239 9101 tablet_copy_source_session.cc:205] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:51.192687 12001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee688e32b7d049509a39259d13439cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:51.195137 12001 tablet_copy_client.cc:783] T ee688e32b7d049509a39259d13439cf6 P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:51.195397 12001 tablet_copy_client.cc:647] T ee688e32b7d049509a39259d13439cf6 P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:51.197494 12001 tablet_copy_client.cc:523] T ee688e32b7d049509a39259d13439cf6 P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:51.199535 12001 tablet_bootstrap.cc:492] T ee688e32b7d049509a39259d13439cf6 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:51.205423 12001 tablet_bootstrap.cc:492] T ee688e32b7d049509a39259d13439cf6 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:51.205740 12001 tablet_bootstrap.cc:492] T ee688e32b7d049509a39259d13439cf6 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap complete.
I20240503 13:59:51.205966 12001 ts_tablet_manager.cc:1367] T ee688e32b7d049509a39259d13439cf6 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 13:59:51.206792 12001 raft_consensus.cc:348] T ee688e32b7d049509a39259d13439cf6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:51.206928 12001 raft_consensus.cc:729] T ee688e32b7d049509a39259d13439cf6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: LEARNER
I20240503 13:59:51.207067 12001 consensus_queue.cc:260] T ee688e32b7d049509a39259d13439cf6 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:51.207499 12001 ts_tablet_manager.cc:1398] T ee688e32b7d049509a39259d13439cf6 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:51.208580 9101 tablet_copy_service.cc:342] P 8d574fb4a89d4eea947c859470d44dd4: Request end of tablet copy session f58065db70b64ef8b1dc499aa6ae4460-ee688e32b7d049509a39259d13439cf6 received from {username='slave'} at 127.0.99.132:53261
I20240503 13:59:51.208674 9101 tablet_copy_service.cc:433] P 8d574fb4a89d4eea947c859470d44dd4: ending tablet copy session f58065db70b64ef8b1dc499aa6ae4460-ee688e32b7d049509a39259d13439cf6 on tablet ee688e32b7d049509a39259d13439cf6 with peer f58065db70b64ef8b1dc499aa6ae4460
I20240503 13:59:51.303198 9081 raft_consensus.cc:1203] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:51.308681 11911 ts_tablet_manager.cc:902] T 62c77ec99cf443d2b7d48b12c8855883 P 4a85e492c8154e9881a9cc2a105d7e64: Initiating tablet copy from peer 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:51.309202 11911 tablet_copy_client.cc:308] T 62c77ec99cf443d2b7d48b12c8855883 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:39481
I20240503 13:59:51.309651 9101 tablet_copy_service.cc:140] P 8d574fb4a89d4eea947c859470d44dd4: Received BeginTabletCopySession request for tablet 62c77ec99cf443d2b7d48b12c8855883 from peer 4a85e492c8154e9881a9cc2a105d7e64 ({username='slave'} at 127.0.99.131:44581)
I20240503 13:59:51.309753 9101 tablet_copy_service.cc:161] P 8d574fb4a89d4eea947c859470d44dd4: Beginning new tablet copy session on tablet 62c77ec99cf443d2b7d48b12c8855883 from peer 4a85e492c8154e9881a9cc2a105d7e64 at {username='slave'} at 127.0.99.131:44581: session id = 4a85e492c8154e9881a9cc2a105d7e64-62c77ec99cf443d2b7d48b12c8855883
I20240503 13:59:51.310659 9101 tablet_copy_source_session.cc:205] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:51.311555 11911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c77ec99cf443d2b7d48b12c8855883. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:51.313835 11911 tablet_copy_client.cc:783] T 62c77ec99cf443d2b7d48b12c8855883 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:51.314002 11911 tablet_copy_client.cc:647] T 62c77ec99cf443d2b7d48b12c8855883 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:51.315989 11911 tablet_copy_client.cc:523] T 62c77ec99cf443d2b7d48b12c8855883 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:51.317153 11911 tablet_bootstrap.cc:492] T 62c77ec99cf443d2b7d48b12c8855883 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:51.321321 6660 raft_consensus.cc:1203] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:51.322108 11911 tablet_bootstrap.cc:492] T 62c77ec99cf443d2b7d48b12c8855883 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:51.322358 11911 tablet_bootstrap.cc:492] T 62c77ec99cf443d2b7d48b12c8855883 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap complete.
I20240503 13:59:51.324179 11911 ts_tablet_manager.cc:1367] T 62c77ec99cf443d2b7d48b12c8855883 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240503 13:59:51.324563 11911 raft_consensus.cc:348] T 62c77ec99cf443d2b7d48b12c8855883 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:51.324668 11911 raft_consensus.cc:729] T 62c77ec99cf443d2b7d48b12c8855883 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: LEARNER
I20240503 13:59:51.324787 11911 consensus_queue.cc:260] T 62c77ec99cf443d2b7d48b12c8855883 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:51.325068 11911 ts_tablet_manager.cc:1398] T 62c77ec99cf443d2b7d48b12c8855883 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:51.325831 9101 tablet_copy_service.cc:342] P 8d574fb4a89d4eea947c859470d44dd4: Request end of tablet copy session 4a85e492c8154e9881a9cc2a105d7e64-62c77ec99cf443d2b7d48b12c8855883 received from {username='slave'} at 127.0.99.131:44581
I20240503 13:59:51.325912 9101 tablet_copy_service.cc:433] P 8d574fb4a89d4eea947c859470d44dd4: ending tablet copy session 4a85e492c8154e9881a9cc2a105d7e64-62c77ec99cf443d2b7d48b12c8855883 on tablet 62c77ec99cf443d2b7d48b12c8855883 with peer 4a85e492c8154e9881a9cc2a105d7e64
I20240503 13:59:51.344141 6660 raft_consensus.cc:1203] T f853508f20d4431bb35e3e05b659953a P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:51.361538 11918 ts_tablet_manager.cc:902] T bccac59f680046389bbe23ff262a82d6 P 5f46e19644fd4f4b825c86ba6af768da: Initiating tablet copy from peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:51.362042 11918 tablet_copy_client.cc:308] T bccac59f680046389bbe23ff262a82d6 P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:34547
I20240503 13:59:51.362607 8968 tablet_copy_service.cc:140] P 06ca8dbc004c42b99faa527a4e6f9545: Received BeginTabletCopySession request for tablet bccac59f680046389bbe23ff262a82d6 from peer 5f46e19644fd4f4b825c86ba6af768da ({username='slave'} at 127.0.99.129:58641)
I20240503 13:59:51.362751 8968 tablet_copy_service.cc:161] P 06ca8dbc004c42b99faa527a4e6f9545: Beginning new tablet copy session on tablet bccac59f680046389bbe23ff262a82d6 from peer 5f46e19644fd4f4b825c86ba6af768da at {username='slave'} at 127.0.99.129:58641: session id = 5f46e19644fd4f4b825c86ba6af768da-bccac59f680046389bbe23ff262a82d6
I20240503 13:59:51.364099 8968 tablet_copy_source_session.cc:205] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:51.365056 11918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bccac59f680046389bbe23ff262a82d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:51.366762 11918 tablet_copy_client.cc:783] T bccac59f680046389bbe23ff262a82d6 P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:51.366946 11918 tablet_copy_client.cc:647] T bccac59f680046389bbe23ff262a82d6 P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:51.368875 11918 tablet_copy_client.cc:523] T bccac59f680046389bbe23ff262a82d6 P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:51.370430 11918 tablet_bootstrap.cc:492] T bccac59f680046389bbe23ff262a82d6 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:51.374419 11918 tablet_bootstrap.cc:492] T bccac59f680046389bbe23ff262a82d6 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:51.374644 11918 tablet_bootstrap.cc:492] T bccac59f680046389bbe23ff262a82d6 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap complete.
I20240503 13:59:51.374792 11918 ts_tablet_manager.cc:1367] T bccac59f680046389bbe23ff262a82d6 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 13:59:51.375242 11918 raft_consensus.cc:348] T bccac59f680046389bbe23ff262a82d6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:51.375451 11918 raft_consensus.cc:729] T bccac59f680046389bbe23ff262a82d6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: LEARNER
I20240503 13:59:51.375566 11918 consensus_queue.cc:260] T bccac59f680046389bbe23ff262a82d6 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:51.375869 11918 ts_tablet_manager.cc:1398] T bccac59f680046389bbe23ff262a82d6 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:51.376570 8968 tablet_copy_service.cc:342] P 06ca8dbc004c42b99faa527a4e6f9545: Request end of tablet copy session 5f46e19644fd4f4b825c86ba6af768da-bccac59f680046389bbe23ff262a82d6 received from {username='slave'} at 127.0.99.129:58641
I20240503 13:59:51.376670 8968 tablet_copy_service.cc:433] P 06ca8dbc004c42b99faa527a4e6f9545: ending tablet copy session 5f46e19644fd4f4b825c86ba6af768da-bccac59f680046389bbe23ff262a82d6 on tablet bccac59f680046389bbe23ff262a82d6 with peer 5f46e19644fd4f4b825c86ba6af768da
I20240503 13:59:51.439594 9342 raft_consensus.cc:1203] T dfa48440020d4754be0f5ddf159c23fe P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:51.447676 9211 raft_consensus.cc:1203] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:51.482690 6791 raft_consensus.cc:1203] T a7e9c80dbf8c41469e54535c20648df4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:51.519681 8946 raft_consensus.cc:1203] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:51.525102 9211 raft_consensus.cc:1203] T 8f57f1ad5c604b5e957374bab029a5df P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:51.559432 6791 raft_consensus.cc:1203] T 7a44f5349cc44be69e9b52f8047d2bb1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:51.589927 6527 raft_consensus.cc:1203] T 8f9149a9faa746cd9542aab02a8e2850 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:51.593827 6660 raft_consensus.cc:1203] T b7e1060596204c60882388e984986ac6 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:51.618547 6660 raft_consensus.cc:1203] T 1eec99c0bb994dd7b6ae63918125bccb P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:51.640750 6791 raft_consensus.cc:1203] T 1350225b1a2848debca68e1bad5528c5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:51.646018 6660 raft_consensus.cc:1203] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:51.648057 9080 raft_consensus.cc:1203] T e2ae8c7089db42199a2e6071800d9efc P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:51.661818 6527 raft_consensus.cc:1203] T 5c7b12e4dc3943a299fc8a49cbc02694 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:51.678597 9080 raft_consensus.cc:1203] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:51.720846 11167 raft_consensus.cc:1050] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6: attempting to promote NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 to VOTER
I20240503 13:59:51.721406 11167 consensus_queue.cc:237] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 13:59:51.722640 6791 raft_consensus.cc:1261] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:51.722640 6660 raft_consensus.cc:1261] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:51.722910 9080 raft_consensus.cc:1261] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:51.723212 11690 consensus_queue.cc:1035] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:51.723562 12172 consensus_queue.cc:1035] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:51.723762 11690 consensus_queue.cc:1035] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:51.726616 11378 raft_consensus.cc:2932] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:51.727078 6660 raft_consensus.cc:2932] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:51.729702 6791 raft_consensus.cc:2932] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:51.730109 6381 catalog_manager.cc:5484] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:51.730638 9080 raft_consensus.cc:2932] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:51.739646 6920 raft_consensus.cc:1203] T ee688e32b7d049509a39259d13439cf6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:51.880162 9343 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "96ce36eb7c25413eaae380f32b465df7"
dest_uuid: "0d77752181e944698e809e325e5937e6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48412
I20240503 13:59:51.880343 9343 raft_consensus.cc:595] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:51.897012 11471 raft_consensus.cc:1050] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735: attempting to promote NON_VOTER 2145d15ab2d74c0287839266f1411a43 to VOTER
I20240503 13:59:51.897533 11471 consensus_queue.cc:237] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 13:59:51.899039 6658 raft_consensus.cc:1261] T f853508f20d4431bb35e3e05b659953a P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:51.899745 8946 raft_consensus.cc:1261] T f853508f20d4431bb35e3e05b659953a P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:51.900121 6527 raft_consensus.cc:1261] T f853508f20d4431bb35e3e05b659953a P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:51.900744 12063 consensus_queue.cc:1035] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:51.900936 12243 consensus_queue.cc:1035] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:51.901185 12063 consensus_queue.cc:1035] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:51.905673 11665 raft_consensus.cc:2932] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:51.908402 8946 raft_consensus.cc:2932] T f853508f20d4431bb35e3e05b659953a P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:51.908887 6527 raft_consensus.cc:2932] T f853508f20d4431bb35e3e05b659953a P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:51.911219 6382 catalog_manager.cc:5484] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:51.912943 6658 raft_consensus.cc:2932] T f853508f20d4431bb35e3e05b659953a P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:51.917443 9211 consensus_queue.cc:237] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 13:59:51.918964 6658 raft_consensus.cc:1261] T f853508f20d4431bb35e3e05b659953a P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:51.919391 6527 raft_consensus.cc:1261] T f853508f20d4431bb35e3e05b659953a P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:51.919747 12243 consensus_queue.cc:1035] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:51.920671 12243 consensus_queue.cc:1035] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:51.929080 12243 raft_consensus.cc:2932] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:51.929344 12063 raft_consensus.cc:1050] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735: attempting to promote NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 to VOTER
I20240503 13:59:51.929713 12063 consensus_queue.cc:237] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 13:59:51.930406 6527 raft_consensus.cc:1203] T bccac59f680046389bbe23ff262a82d6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:51.931033 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f853508f20d4431bb35e3e05b659953a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:51.931330 6527 raft_consensus.cc:1261] T a7e9c80dbf8c41469e54535c20648df4 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:51.931682 8946 raft_consensus.cc:1261] T a7e9c80dbf8c41469e54535c20648df4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:51.932325 6791 raft_consensus.cc:1261] T a7e9c80dbf8c41469e54535c20648df4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:51.932775 11665 consensus_queue.cc:1035] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:51.933451 6791 raft_consensus.cc:1203] T 62c77ec99cf443d2b7d48b12c8855883 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:51.933846 6527 raft_consensus.cc:2932] T f853508f20d4431bb35e3e05b659953a P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:51.935098 6660 raft_consensus.cc:2932] T f853508f20d4431bb35e3e05b659953a P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:51.935449 11471 consensus_queue.cc:1035] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:51.935787 11471 consensus_queue.cc:1035] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:51.936594 6382 catalog_manager.cc:5484] T f853508f20d4431bb35e3e05b659953a P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:51.939379 11311 raft_consensus.cc:1050] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460: attempting to promote NON_VOTER 0d77752181e944698e809e325e5937e6 to VOTER
I20240503 13:59:51.939920 11311 consensus_queue.cc:237] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 13:59:51.942006 9080 raft_consensus.cc:1261] T dfa48440020d4754be0f5ddf159c23fe P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:51.942111 12243 raft_consensus.cc:2932] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:51.943619 12069 consensus_queue.cc:1035] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:51.945744 9343 raft_consensus.cc:1261] T dfa48440020d4754be0f5ddf159c23fe P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:51.945886 6527 raft_consensus.cc:2932] T a7e9c80dbf8c41469e54535c20648df4 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:51.947990 12266 consensus_queue.cc:1035] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 13:59:51.949441 6790 raft_consensus.cc:2932] T a7e9c80dbf8c41469e54535c20648df4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:51.949903 6658 raft_consensus.cc:1261] T dfa48440020d4754be0f5ddf159c23fe P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:51.950412 12069 consensus_queue.cc:1035] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:51.950686 11690 raft_consensus.cc:1050] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6: attempting to promote NON_VOTER 2145d15ab2d74c0287839266f1411a43 to VOTER
I20240503 13:59:51.951063 11690 consensus_queue.cc:237] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 13:59:51.952579 8948 raft_consensus.cc:1261] T 0fa4727297da40eb9f58f5daa8150e0e P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:51.952948 12172 consensus_queue.cc:1035] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:51.953469 6527 raft_consensus.cc:1261] T 0fa4727297da40eb9f58f5daa8150e0e P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:51.954633 11378 consensus_queue.cc:1035] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:51.954540 6384 catalog_manager.cc:5484] T a7e9c80dbf8c41469e54535c20648df4 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:51.954826 8946 raft_consensus.cc:2932] T a7e9c80dbf8c41469e54535c20648df4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:51.956071 6658 raft_consensus.cc:1261] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:51.957844 12172 consensus_queue.cc:1035] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:51.961583 12172 raft_consensus.cc:2932] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:51.962852 8948 raft_consensus.cc:2932] T 0fa4727297da40eb9f58f5daa8150e0e P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:51.965626 8926 tablet_service.cc:1514] Processing DeleteTablet for tablet f853508f20d4431bb35e3e05b659953a with delete_type TABLET_DATA_TOMBSTONED (TS 06ca8dbc004c42b99faa527a4e6f9545 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:51.966198 6658 raft_consensus.cc:2932] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:51.966907 6381 catalog_manager.cc:5484] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:51.968725 6527 raft_consensus.cc:2932] T 0fa4727297da40eb9f58f5daa8150e0e P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:51.971774 12172 raft_consensus.cc:1050] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6: attempting to promote NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 to VOTER
I20240503 13:59:51.972258 12172 consensus_queue.cc:237] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
I20240503 13:59:51.973085 9342 consensus_queue.cc:237] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 13:59:51.976298 12279 tablet_replica.cc:330] T f853508f20d4431bb35e3e05b659953a P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:51.976457 12279 raft_consensus.cc:2229] T f853508f20d4431bb35e3e05b659953a P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:51.976634 12279 raft_consensus.cc:2258] T f853508f20d4431bb35e3e05b659953a P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:51.978010 12279 ts_tablet_manager.cc:1875] T f853508f20d4431bb35e3e05b659953a P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:51.980420 12279 ts_tablet_manager.cc:1888] T f853508f20d4431bb35e3e05b659953a P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:51.980490 12279 log.cc:1196] T f853508f20d4431bb35e3e05b659953a P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/f853508f20d4431bb35e3e05b659953a
I20240503 13:59:51.981127 9211 consensus_queue.cc:237] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 13:59:51.982646 6791 raft_consensus.cc:1261] T a7e9c80dbf8c41469e54535c20648df4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:51.982758 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet f853508f20d4431bb35e3e05b659953a (table rebalancer_extra_table_10 [id=b511123c500546da8d5ca21cb43ec824]) successfully deleted
I20240503 13:59:51.983047 11471 consensus_queue.cc:1035] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:51.983692 6660 raft_consensus.cc:1261] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:51.983716 8948 raft_consensus.cc:1261] T a7e9c80dbf8c41469e54535c20648df4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:51.984328 11690 consensus_queue.cc:1035] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:51.984625 11471 consensus_queue.cc:1035] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:51.985507 8948 raft_consensus.cc:1261] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:51.986073 6527 raft_consensus.cc:1261] T 0fa4727297da40eb9f58f5daa8150e0e P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:51.986141 12063 raft_consensus.cc:2932] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:51.986507 8946 raft_consensus.cc:2932] T a7e9c80dbf8c41469e54535c20648df4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:51.988315 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a7e9c80dbf8c41469e54535c20648df4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:51.989183 9211 raft_consensus.cc:1261] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:51.989571 6791 raft_consensus.cc:2932] T a7e9c80dbf8c41469e54535c20648df4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:51.991391 6381 catalog_manager.cc:5484] T a7e9c80dbf8c41469e54535c20648df4 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index 3 to 4, VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:51.992723 9081 raft_consensus.cc:1261] T 0fe53d65ab4d482aad118d6e3bd5533c P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:51.995512 6507 tablet_service.cc:1514] Processing DeleteTablet for tablet a7e9c80dbf8c41469e54535c20648df4 with delete_type TABLET_DATA_TOMBSTONED (TS 5f46e19644fd4f4b825c86ba6af768da not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:51.995671 11690 raft_consensus.cc:2932] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:51.996024 11378 consensus_queue.cc:1035] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:51.996595 12292 tablet_replica.cc:330] T a7e9c80dbf8c41469e54535c20648df4 P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:51.996764 12292 raft_consensus.cc:2229] T a7e9c80dbf8c41469e54535c20648df4 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:51.996955 12292 raft_consensus.cc:2258] T a7e9c80dbf8c41469e54535c20648df4 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:51.997745 12292 ts_tablet_manager.cc:1875] T a7e9c80dbf8c41469e54535c20648df4 P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:51.998589 6374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0fa4727297da40eb9f58f5daa8150e0e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:51.998821 6527 raft_consensus.cc:2932] T 0fa4727297da40eb9f58f5daa8150e0e P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:52.000104 6381 catalog_manager.cc:5484] T 0fa4727297da40eb9f58f5daa8150e0e P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:52.000771 12292 ts_tablet_manager.cc:1888] T a7e9c80dbf8c41469e54535c20648df4 P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:52.000871 12292 log.cc:1196] T a7e9c80dbf8c41469e54535c20648df4 P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/a7e9c80dbf8c41469e54535c20648df4
I20240503 13:59:52.001631 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet a7e9c80dbf8c41469e54535c20648df4 (table rebalancer_extra_table_12 [id=3fa9362ac0d94323a9cd7a56209c3e07]) successfully deleted
I20240503 13:59:52.005988 8926 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fa4727297da40eb9f58f5daa8150e0e with delete_type TABLET_DATA_TOMBSTONED (TS 06ca8dbc004c42b99faa527a4e6f9545 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:52.006217 12279 tablet_replica.cc:330] T 0fa4727297da40eb9f58f5daa8150e0e P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:52.006372 12279 raft_consensus.cc:2229] T 0fa4727297da40eb9f58f5daa8150e0e P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:52.006632 12279 raft_consensus.cc:2258] T 0fa4727297da40eb9f58f5daa8150e0e P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:52.007267 12279 ts_tablet_manager.cc:1875] T 0fa4727297da40eb9f58f5daa8150e0e P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:52.008246 11167 consensus_queue.cc:1035] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.008543 11378 consensus_queue.cc:1035] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.008766 11690 consensus_queue.cc:1035] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.009558 12279 ts_tablet_manager.cc:1888] T 0fa4727297da40eb9f58f5daa8150e0e P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:52.009631 12279 log.cc:1196] T 0fa4727297da40eb9f58f5daa8150e0e P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/0fa4727297da40eb9f58f5daa8150e0e
I20240503 13:59:52.010380 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 0fa4727297da40eb9f58f5daa8150e0e (table rebalancer_extra_table_16 [id=82c39415abdc4fe8a6c8d9f312c84cbc]) successfully deleted
I20240503 13:59:52.010802 12266 raft_consensus.cc:2932] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:52.011628 9080 raft_consensus.cc:2932] T dfa48440020d4754be0f5ddf159c23fe P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:52.012889 6658 raft_consensus.cc:2932] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:52.013671 9343 raft_consensus.cc:2932] T dfa48440020d4754be0f5ddf159c23fe P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:52.017686 11311 raft_consensus.cc:1050] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460: attempting to promote NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 to VOTER
I20240503 13:59:52.018110 11311 consensus_queue.cc:237] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 13:59:52.018015 6384 catalog_manager.cc:5484] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:52.020195 6790 raft_consensus.cc:1261] T 7a44f5349cc44be69e9b52f8047d2bb1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.020705 11271 consensus_queue.cc:1035] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.020807 9205 raft_consensus.cc:1261] T 7a44f5349cc44be69e9b52f8047d2bb1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.021253 6920 consensus_queue.cc:237] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 13:59:52.021364 6658 raft_consensus.cc:2932] T dfa48440020d4754be0f5ddf159c23fe P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:52.021472 11271 consensus_queue.cc:1035] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.024338 6658 raft_consensus.cc:1261] T dfa48440020d4754be0f5ddf159c23fe P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.026965 12266 consensus_queue.cc:1035] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.028224 11378 raft_consensus.cc:2932] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:52.029156 12069 raft_consensus.cc:2932] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:52.030834 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dfa48440020d4754be0f5ddf159c23fe with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:52.031852 6660 raft_consensus.cc:2932] T dfa48440020d4754be0f5ddf159c23fe P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:52.033176 6382 catalog_manager.cc:5484] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index 2 to 3, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:52.033876 9343 raft_consensus.cc:1261] T 7a44f5349cc44be69e9b52f8047d2bb1 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.034274 9343 raft_consensus.cc:1261] T dfa48440020d4754be0f5ddf159c23fe P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.034502 6384 catalog_manager.cc:5484] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:52.034833 8946 raft_consensus.cc:2932] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:52.038703 9060 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa48440020d4754be0f5ddf159c23fe with delete_type TABLET_DATA_TOMBSTONED (TS 8d574fb4a89d4eea947c859470d44dd4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:52.039324 12063 raft_consensus.cc:1050] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735: attempting to promote NON_VOTER 2145d15ab2d74c0287839266f1411a43 to VOTER
I20240503 13:59:52.039736 12063 consensus_queue.cc:237] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 13:59:52.041102 11690 raft_consensus.cc:1050] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6: attempting to promote NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 to VOTER
I20240503 13:59:52.041129 6660 raft_consensus.cc:1261] T b7e1060596204c60882388e984986ac6 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.042318 6791 raft_consensus.cc:1261] T b7e1060596204c60882388e984986ac6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.042740 12308 tablet_replica.cc:330] T dfa48440020d4754be0f5ddf159c23fe P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:52.042806 11665 consensus_queue.cc:1035] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.042897 12308 raft_consensus.cc:2229] T dfa48440020d4754be0f5ddf159c23fe P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:52.043035 12308 raft_consensus.cc:2258] T dfa48440020d4754be0f5ddf159c23fe P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:52.044948 11471 consensus_queue.cc:1035] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.045940 9342 raft_consensus.cc:1261] T b7e1060596204c60882388e984986ac6 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.047224 12243 consensus_queue.cc:1035] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.047224 11665 raft_consensus.cc:2932] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:52.047824 11690 consensus_queue.cc:237] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 13:59:52.049504 6789 raft_consensus.cc:1261] T 1350225b1a2848debca68e1bad5528c5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.050792 11271 consensus_queue.cc:1035] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.051179 11271 consensus_queue.cc:1035] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.052006 11378 consensus_queue.cc:1035] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.052095 6527 raft_consensus.cc:1261] T 1350225b1a2848debca68e1bad5528c5 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.052379 9080 raft_consensus.cc:2932] T 0fe53d65ab4d482aad118d6e3bd5533c P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:52.053542 9342 raft_consensus.cc:2932] T dfa48440020d4754be0f5ddf159c23fe P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:52.053606 11378 consensus_queue.cc:1035] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.054839 9210 raft_consensus.cc:2932] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:52.055058 12308 ts_tablet_manager.cc:1875] T dfa48440020d4754be0f5ddf159c23fe P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:52.055069 11690 raft_consensus.cc:1050] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6: attempting to promote NON_VOTER 2145d15ab2d74c0287839266f1411a43 to VOTER
I20240503 13:59:52.055447 11690 consensus_queue.cc:237] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 13:59:52.057356 12308 ts_tablet_manager.cc:1888] T dfa48440020d4754be0f5ddf159c23fe P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:52.057430 12308 log.cc:1196] T dfa48440020d4754be0f5ddf159c23fe P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/dfa48440020d4754be0f5ddf159c23fe
I20240503 13:59:52.057456 6660 raft_consensus.cc:1261] T 1eec99c0bb994dd7b6ae63918125bccb P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.058173 6920 raft_consensus.cc:1261] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.058404 11167 consensus_queue.cc:1035] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.058945 8946 raft_consensus.cc:1261] T 1eec99c0bb994dd7b6ae63918125bccb P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.059252 11584 raft_consensus.cc:1050] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43: attempting to promote NON_VOTER 5f46e19644fd4f4b825c86ba6af768da to VOTER
I20240503 13:59:52.059758 9205 raft_consensus.cc:1261] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.059753 11584 consensus_queue.cc:237] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 13:59:52.059880 11271 raft_consensus.cc:2932] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:52.060590 12314 consensus_queue.cc:1035] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.061538 6790 raft_consensus.cc:1261] T 8f9149a9faa746cd9542aab02a8e2850 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.061918 9080 raft_consensus.cc:1261] T 8f9149a9faa746cd9542aab02a8e2850 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.062153 6528 raft_consensus.cc:1261] T 8f9149a9faa746cd9542aab02a8e2850 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.062546 11320 consensus_queue.cc:1035] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.062708 12321 consensus_queue.cc:1035] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.062767 9211 raft_consensus.cc:2932] T 7a44f5349cc44be69e9b52f8047d2bb1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:52.063014 11320 consensus_queue.cc:1035] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.063174 6790 raft_consensus.cc:2932] T 7a44f5349cc44be69e9b52f8047d2bb1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:52.065201 12314 consensus_queue.cc:1035] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.065920 12314 raft_consensus.cc:2932] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:52.066236 6788 raft_consensus.cc:2932] T 1350225b1a2848debca68e1bad5528c5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:52.066619 6384 catalog_manager.cc:5484] T 7a44f5349cc44be69e9b52f8047d2bb1 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:52.068608 6527 raft_consensus.cc:2932] T 1350225b1a2848debca68e1bad5528c5 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:52.069594 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet dfa48440020d4754be0f5ddf159c23fe (table rebalancer_extra_table_64 [id=06e71db014ed4195a64a38b339ee4ccb]) successfully deleted
I20240503 13:59:52.070574 11378 raft_consensus.cc:2932] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:52.072253 12314 consensus_queue.cc:1035] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.072453 9205 raft_consensus.cc:2932] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:52.072834 11320 raft_consensus.cc:2932] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:52.071203 6381 catalog_manager.cc:5484] T 1350225b1a2848debca68e1bad5528c5 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:52.074312 6660 raft_consensus.cc:2932] T 1eec99c0bb994dd7b6ae63918125bccb P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:52.074982 6527 raft_consensus.cc:2932] T 8f9149a9faa746cd9542aab02a8e2850 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:52.075943 6660 raft_consensus.cc:2932] T b7e1060596204c60882388e984986ac6 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:52.076647 6920 raft_consensus.cc:2932] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:52.077816 6387 catalog_manager.cc:5484] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:52.078451 6791 raft_consensus.cc:2932] T b7e1060596204c60882388e984986ac6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:52.078441 9080 raft_consensus.cc:2932] T 8f9149a9faa746cd9542aab02a8e2850 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:52.079159 6385 catalog_manager.cc:5484] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:52.080672 9343 raft_consensus.cc:2932] T b7e1060596204c60882388e984986ac6 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:52.080011 6791 raft_consensus.cc:2932] T 8f9149a9faa746cd9542aab02a8e2850 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:52.082337 11271 raft_consensus.cc:1050] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460: attempting to promote NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 to VOTER
I20240503 13:59:52.082732 11271 consensus_queue.cc:237] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 13:59:52.082953 9342 raft_consensus.cc:2932] T 7a44f5349cc44be69e9b52f8047d2bb1 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:52.083693 6922 consensus_queue.cc:237] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 13:59:52.085033 6790 raft_consensus.cc:1261] T 7a44f5349cc44be69e9b52f8047d2bb1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.085630 11311 consensus_queue.cc:1035] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.086216 9211 raft_consensus.cc:1261] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:52.086702 12266 consensus_queue.cc:1035] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:52.088491 9341 raft_consensus.cc:1261] T 7a44f5349cc44be69e9b52f8047d2bb1 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.088374 11311 raft_consensus.cc:2932] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:52.089989 6790 raft_consensus.cc:2932] T 7a44f5349cc44be69e9b52f8047d2bb1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:52.089987 8946 raft_consensus.cc:2932] T 1eec99c0bb994dd7b6ae63918125bccb P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:52.091558 9342 consensus_queue.cc:237] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
I20240503 13:59:52.092012 12069 consensus_queue.cc:1035] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.092077 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7a44f5349cc44be69e9b52f8047d2bb1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:52.093825 8946 raft_consensus.cc:1261] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.093854 9211 raft_consensus.cc:1261] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.094256 12314 consensus_queue.cc:1035] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.094727 6660 raft_consensus.cc:1261] T 350ba659df804b95b0692091dc2f68a2 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:52.095139 6791 raft_consensus.cc:1261] T 350ba659df804b95b0692091dc2f68a2 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:52.095170 11378 consensus_queue.cc:1035] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.095485 12266 consensus_queue.cc:1035] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:52.095815 12266 consensus_queue.cc:1035] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:52.096719 9211 consensus_queue.cc:237] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 13:59:52.098224 9342 raft_consensus.cc:2932] T 7a44f5349cc44be69e9b52f8047d2bb1 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:52.098321 12172 raft_consensus.cc:2932] T 0fe53d65ab4d482aad118d6e3bd5533c P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:52.099021 6379 catalog_manager.cc:5484] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:52.099598 6374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0fe53d65ab4d482aad118d6e3bd5533c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:52.099840 9205 raft_consensus.cc:2932] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:52.100037 6790 raft_consensus.cc:1261] T b7e1060596204c60882388e984986ac6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.100390 11471 consensus_queue.cc:1035] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.101277 6658 raft_consensus.cc:1261] T b7e1060596204c60882388e984986ac6 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.101702 11471 consensus_queue.cc:1035] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.101927 6381 catalog_manager.cc:5484] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:52.102213 8946 raft_consensus.cc:2932] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:52.102974 12322 raft_consensus.cc:1050] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43: attempting to promote NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 to VOTER
I20240503 13:59:52.103435 12322 consensus_queue.cc:237] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 13:59:52.104398 11471 raft_consensus.cc:2932] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:52.105391 6922 raft_consensus.cc:1261] T 8f57f1ad5c604b5e957374bab029a5df P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:52.105718 12321 consensus_queue.cc:1035] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:52.106303 6788 raft_consensus.cc:2932] T b7e1060596204c60882388e984986ac6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:52.106416 6658 raft_consensus.cc:2932] T b7e1060596204c60882388e984986ac6 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:52.107674 9211 raft_consensus.cc:1261] T 8f57f1ad5c604b5e957374bab029a5df P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:52.107847 9341 consensus_queue.cc:237] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 13:59:52.108932 12321 consensus_queue.cc:1035] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:52.109126 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b7e1060596204c60882388e984986ac6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:52.109180 6658 raft_consensus.cc:1261] T 1eec99c0bb994dd7b6ae63918125bccb P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.110644 12172 consensus_queue.cc:1035] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.110865 9205 raft_consensus.cc:1261] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.111084 11167 raft_consensus.cc:1050] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6: attempting to promote NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 to VOTER
I20240503 13:59:52.111476 11167 consensus_queue.cc:237] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 13:59:52.112186 6386 catalog_manager.cc:5484] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:52.112627 9192 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a44f5349cc44be69e9b52f8047d2bb1 with delete_type TABLET_DATA_TOMBSTONED (TS ab1d21935d6049888eb0ceccc3c9d735 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57960
I20240503 13:59:52.113054 11690 raft_consensus.cc:2932] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:52.115027 6374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1eec99c0bb994dd7b6ae63918125bccb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:52.115247 9080 raft_consensus.cc:1261] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.115227 11167 consensus_queue.cc:1035] T 1eec99c0bb994dd7b6ae63918125bccb P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.116251 6385 catalog_manager.cc:5484] T b7e1060596204c60882388e984986ac6 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:52.116927 6527 raft_consensus.cc:1261] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.117009 6921 raft_consensus.cc:1261] T 3d9906a118744dd6a78d50e52f1961e2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.117798 6657 consensus_queue.cc:237] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 13:59:52.118309 6790 raft_consensus.cc:1261] T 8f57f1ad5c604b5e957374bab029a5df P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:52.119061 11167 consensus_queue.cc:1035] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.119083 11584 consensus_queue.cc:1035] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:52.119402 11167 consensus_queue.cc:1035] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.119709 11167 consensus_queue.cc:1035] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.123919 9211 raft_consensus.cc:2932] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:52.126307 6657 raft_consensus.cc:2932] T 1eec99c0bb994dd7b6ae63918125bccb P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:52.127207 6526 raft_consensus.cc:1261] T 8f9149a9faa746cd9542aab02a8e2850 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.127748 9080 raft_consensus.cc:1261] T 8f9149a9faa746cd9542aab02a8e2850 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.128580 9060 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe53d65ab4d482aad118d6e3bd5533c with delete_type TABLET_DATA_TOMBSTONED (TS 8d574fb4a89d4eea947c859470d44dd4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:52.128527 11690 raft_consensus.cc:2932] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:52.128793 12308 tablet_replica.cc:330] T 0fe53d65ab4d482aad118d6e3bd5533c P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:52.128932 12308 raft_consensus.cc:2229] T 0fe53d65ab4d482aad118d6e3bd5533c P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:52.128798 11311 raft_consensus.cc:2932] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:52.129050 12308 raft_consensus.cc:2258] T 0fe53d65ab4d482aad118d6e3bd5533c P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:52.128894 6528 raft_consensus.cc:2932] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:52.130347 9080 raft_consensus.cc:2932] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:52.130951 12308 ts_tablet_manager.cc:1875] T 0fe53d65ab4d482aad118d6e3bd5533c P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:52.130878 6921 raft_consensus.cc:2932] T 3d9906a118744dd6a78d50e52f1961e2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:52.132757 9211 raft_consensus.cc:2932] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:52.133263 12308 ts_tablet_manager.cc:1888] T 0fe53d65ab4d482aad118d6e3bd5533c P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:52.133338 12308 log.cc:1196] T 0fe53d65ab4d482aad118d6e3bd5533c P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/0fe53d65ab4d482aad118d6e3bd5533c
I20240503 13:59:52.134634 6386 catalog_manager.cc:5484] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:52.135066 12339 tablet_replica.cc:330] T 7a44f5349cc44be69e9b52f8047d2bb1 P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 13:59:52.135216 12339 raft_consensus.cc:2229] T 7a44f5349cc44be69e9b52f8047d2bb1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:52.135408 12339 raft_consensus.cc:2258] T 7a44f5349cc44be69e9b52f8047d2bb1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:52.135596 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet 0fe53d65ab4d482aad118d6e3bd5533c (table rebalancer_extra_table_40 [id=9ae2d84be5cd4571b93dd93183bc33dd]) successfully deleted
I20240503 13:59:52.136201 12339 ts_tablet_manager.cc:1875] T 7a44f5349cc44be69e9b52f8047d2bb1 P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:52.136735 9321 tablet_service.cc:1514] Processing DeleteTablet for tablet b7e1060596204c60882388e984986ac6 with delete_type TABLET_DATA_TOMBSTONED (TS 0d77752181e944698e809e325e5937e6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:52.137486 6380 catalog_manager.cc:5484] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:52.128307 12328 consensus_queue.cc:1035] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20240503 13:59:52.138203 11320 consensus_queue.cc:1035] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.139995 6387 catalog_manager.cc:5484] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:52.140086 6791 raft_consensus.cc:2932] T 350ba659df804b95b0692091dc2f68a2 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:52.141731 11320 raft_consensus.cc:2932] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:52.143544 8926 tablet_service.cc:1514] Processing DeleteTablet for tablet 1eec99c0bb994dd7b6ae63918125bccb with delete_type TABLET_DATA_TOMBSTONED (TS 06ca8dbc004c42b99faa527a4e6f9545 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:52.143749 12279 tablet_replica.cc:330] T 1eec99c0bb994dd7b6ae63918125bccb P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:52.143886 12279 raft_consensus.cc:2229] T 1eec99c0bb994dd7b6ae63918125bccb P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:52.144043 12279 raft_consensus.cc:2258] T 1eec99c0bb994dd7b6ae63918125bccb P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:52.130050 6660 raft_consensus.cc:2932] T 350ba659df804b95b0692091dc2f68a2 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:52.145421 6374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f9149a9faa746cd9542aab02a8e2850 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:52.145608 12279 ts_tablet_manager.cc:1875] T 1eec99c0bb994dd7b6ae63918125bccb P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:52.147629 12063 raft_consensus.cc:1050] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735: attempting to promote NON_VOTER 5f46e19644fd4f4b825c86ba6af768da to VOTER
I20240503 13:59:52.147714 12339 ts_tablet_manager.cc:1888] T 7a44f5349cc44be69e9b52f8047d2bb1 P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:52.147712 12279 ts_tablet_manager.cc:1888] T 1eec99c0bb994dd7b6ae63918125bccb P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:52.147778 12279 log.cc:1196] T 1eec99c0bb994dd7b6ae63918125bccb P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/1eec99c0bb994dd7b6ae63918125bccb
I20240503 13:59:52.147778 12339 log.cc:1196] T 7a44f5349cc44be69e9b52f8047d2bb1 P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/7a44f5349cc44be69e9b52f8047d2bb1
I20240503 13:59:52.148113 12063 consensus_queue.cc:237] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 13:59:52.148670 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 1eec99c0bb994dd7b6ae63918125bccb (table rebalancer_extra_table_38 [id=800413c4c03946b7bfef618ffc28c99e]) successfully deleted
I20240503 13:59:52.149195 9343 consensus_queue.cc:237] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 13:59:52.149786 6921 raft_consensus.cc:1261] T 5c7b12e4dc3943a299fc8a49cbc02694 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.150003 12350 tablet_replica.cc:330] T b7e1060596204c60882388e984986ac6 P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:52.150136 12350 raft_consensus.cc:2229] T b7e1060596204c60882388e984986ac6 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:52.150269 12350 raft_consensus.cc:2258] T b7e1060596204c60882388e984986ac6 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:52.150429 11471 consensus_queue.cc:1035] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.150889 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet 7a44f5349cc44be69e9b52f8047d2bb1 (table rebalancer_extra_table_18 [id=67fda36210c64eee9922da159b8c91a6]) successfully deleted
I20240503 13:59:52.151543 6788 raft_consensus.cc:1261] T 5c7b12e4dc3943a299fc8a49cbc02694 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.152210 6526 raft_consensus.cc:1261] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.152402 11471 consensus_queue.cc:1035] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.152567 11690 consensus_queue.cc:1035] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.153841 6386 catalog_manager.cc:5484] T 8f9149a9faa746cd9542aab02a8e2850 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index 3 to 4, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:52.153946 9081 raft_consensus.cc:2932] T 8f9149a9faa746cd9542aab02a8e2850 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:52.155705 9080 raft_consensus.cc:1261] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.156841 6528 raft_consensus.cc:1261] T 5c7b12e4dc3943a299fc8a49cbc02694 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.157115 6769 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f9149a9faa746cd9542aab02a8e2850 with delete_type TABLET_DATA_TOMBSTONED (TS 4a85e492c8154e9881a9cc2a105d7e64 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:52.157626 11471 consensus_queue.cc:1035] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.158149 12314 consensus_queue.cc:1035] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.161633 6526 raft_consensus.cc:2932] T 8f9149a9faa746cd9542aab02a8e2850 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:52.161803 12314 raft_consensus.cc:2932] T 3d9906a118744dd6a78d50e52f1961e2 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:52.162614 9080 raft_consensus.cc:2932] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:52.165279 6374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3d9906a118744dd6a78d50e52f1961e2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:52.166087 6527 raft_consensus.cc:2932] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:52.166684 12350 ts_tablet_manager.cc:1875] T b7e1060596204c60882388e984986ac6 P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:52.167531 6381 catalog_manager.cc:5484] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index 3 to 4, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:52.165944 11471 raft_consensus.cc:2932] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:52.173391 6921 raft_consensus.cc:2932] T 5c7b12e4dc3943a299fc8a49cbc02694 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:52.174582 6900 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d9906a118744dd6a78d50e52f1961e2 with delete_type TABLET_DATA_TOMBSTONED (TS f58065db70b64ef8b1dc499aa6ae4460 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:52.176375 6527 raft_consensus.cc:2932] T 5c7b12e4dc3943a299fc8a49cbc02694 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:52.176538 6387 catalog_manager.cc:5484] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:52.176828 6790 raft_consensus.cc:2932] T 5c7b12e4dc3943a299fc8a49cbc02694 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:52.180858 12350 ts_tablet_manager.cc:1888] T b7e1060596204c60882388e984986ac6 P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:52.180866 9211 consensus_queue.cc:237] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 13:59:52.181186 12350 log.cc:1196] T b7e1060596204c60882388e984986ac6 P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/b7e1060596204c60882388e984986ac6
I20240503 13:59:52.181804 12357 tablet_replica.cc:330] T 3d9906a118744dd6a78d50e52f1961e2 P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:52.181990 12357 raft_consensus.cc:2229] T 3d9906a118744dd6a78d50e52f1961e2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:52.182194 12357 raft_consensus.cc:2258] T 3d9906a118744dd6a78d50e52f1961e2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:52.182372 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet b7e1060596204c60882388e984986ac6 (table rebalancer_extra_table_42 [id=2c584aa1fb5a4bc4a40f13eaf00c9ddf]) successfully deleted
I20240503 13:59:52.182652 6527 raft_consensus.cc:1261] T 5c7b12e4dc3943a299fc8a49cbc02694 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.182996 12243 consensus_queue.cc:1035] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.183094 12357 ts_tablet_manager.cc:1875] T 3d9906a118744dd6a78d50e52f1961e2 P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:52.184109 6791 raft_consensus.cc:1261] T 5c7b12e4dc3943a299fc8a49cbc02694 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.184878 12243 consensus_queue.cc:1035] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.185880 12357 ts_tablet_manager.cc:1888] T 3d9906a118744dd6a78d50e52f1961e2 P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:52.186100 12357 log.cc:1196] T 3d9906a118744dd6a78d50e52f1961e2 P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/3d9906a118744dd6a78d50e52f1961e2
I20240503 13:59:52.186770 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet 3d9906a118744dd6a78d50e52f1961e2 (table rebalancer_extra_table_52 [id=17d62e19cb344d018bc5ae78ffca2185]) successfully deleted
I20240503 13:59:52.191268 11665 raft_consensus.cc:2932] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:52.193399 6528 raft_consensus.cc:2932] T 5c7b12e4dc3943a299fc8a49cbc02694 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:52.195235 6386 catalog_manager.cc:5484] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index 3 to 4, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:52.195664 6791 raft_consensus.cc:2932] T 5c7b12e4dc3943a299fc8a49cbc02694 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:52.195892 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5c7b12e4dc3943a299fc8a49cbc02694 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:52.197264 12355 tablet_replica.cc:330] T 8f9149a9faa746cd9542aab02a8e2850 P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:52.197427 12355 raft_consensus.cc:2229] T 8f9149a9faa746cd9542aab02a8e2850 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:52.199529 12355 raft_consensus.cc:2258] T 8f9149a9faa746cd9542aab02a8e2850 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:52.201258 6900 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c7b12e4dc3943a299fc8a49cbc02694 with delete_type TABLET_DATA_TOMBSTONED (TS f58065db70b64ef8b1dc499aa6ae4460 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:52.201659 12355 ts_tablet_manager.cc:1875] T 8f9149a9faa746cd9542aab02a8e2850 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:52.201802 12357 tablet_replica.cc:330] T 5c7b12e4dc3943a299fc8a49cbc02694 P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:52.201941 12357 raft_consensus.cc:2229] T 5c7b12e4dc3943a299fc8a49cbc02694 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:52.202057 12357 raft_consensus.cc:2258] T 5c7b12e4dc3943a299fc8a49cbc02694 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:52.202370 12328 raft_consensus.cc:2932] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:52.202863 12357 ts_tablet_manager.cc:1875] T 5c7b12e4dc3943a299fc8a49cbc02694 P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:52.205180 12357 ts_tablet_manager.cc:1888] T 5c7b12e4dc3943a299fc8a49cbc02694 P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:52.205319 12357 log.cc:1196] T 5c7b12e4dc3943a299fc8a49cbc02694 P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/5c7b12e4dc3943a299fc8a49cbc02694
I20240503 13:59:52.205453 12355 ts_tablet_manager.cc:1888] T 8f9149a9faa746cd9542aab02a8e2850 P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:52.205543 12355 log.cc:1196] T 8f9149a9faa746cd9542aab02a8e2850 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/8f9149a9faa746cd9542aab02a8e2850
I20240503 13:59:52.205914 6922 raft_consensus.cc:2932] T 8f57f1ad5c604b5e957374bab029a5df P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:52.206099 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet 8f9149a9faa746cd9542aab02a8e2850 (table rebalancer_extra_table_56 [id=bfd5a93b6dc141158909aa98a0ef7189]) successfully deleted
I20240503 13:59:52.206912 9211 raft_consensus.cc:2932] T 8f57f1ad5c604b5e957374bab029a5df P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:52.207765 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet 5c7b12e4dc3943a299fc8a49cbc02694 (table rebalancer_extra_table_74 [id=a17de5eddcbb412081dd0148ee6471c6]) successfully deleted
I20240503 13:59:52.208518 6790 raft_consensus.cc:2932] T 8f57f1ad5c604b5e957374bab029a5df P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:52.209282 6387 catalog_manager.cc:5484] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:52.212723 11167 raft_consensus.cc:979] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6: : Instructing follower 4a85e492c8154e9881a9cc2a105d7e64 to start an election
I20240503 13:59:52.212877 11690 raft_consensus.cc:1067] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Signalling peer 4a85e492c8154e9881a9cc2a105d7e64 to start an election
I20240503 13:59:52.213835 6790 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "96ce36eb7c25413eaae380f32b465df7"
dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
from {username='slave'} at 127.0.99.136:34769
I20240503 13:59:52.214095 6790 raft_consensus.cc:482] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:52.214267 6790 raft_consensus.cc:3037] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:52.214174 6660 consensus_queue.cc:237] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 13:59:52.215337 6790 raft_consensus.cc:504] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 13:59:52.215526 6791 raft_consensus.cc:1261] T 8f57f1ad5c604b5e957374bab029a5df P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:52.215979 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ce36eb7c25413eaae380f32b465df7" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:52.216104 6660 raft_consensus.cc:3037] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:52.216570 11584 consensus_queue.cc:1035] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:52.217173 6660 raft_consensus.cc:2454] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 2.
I20240503 13:59:52.217505 9211 raft_consensus.cc:1261] T 8f57f1ad5c604b5e957374bab029a5df P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:52.218021 11584 consensus_queue.cc:1035] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:52.218600 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ce36eb7c25413eaae380f32b465df7" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:52.218591 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ce36eb7c25413eaae380f32b465df7" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:52.218735 9080 raft_consensus.cc:3037] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:52.218758 9343 raft_consensus.cc:3032] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:52.218814 9343 raft_consensus.cc:729] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:52.218976 9343 consensus_queue.cc:260] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 13:59:52.219214 9343 raft_consensus.cc:3037] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:52.219424 6790 leader_election.cc:288] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 2 election: Requested vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:52.219771 9080 raft_consensus.cc:2454] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 2.
I20240503 13:59:52.220063 6726 leader_election.cc:302] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 4a85e492c8154e9881a9cc2a105d7e64, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 13:59:52.220290 12032 raft_consensus.cc:2781] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:52.220292 9343 raft_consensus.cc:2454] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 2.
I20240503 13:59:52.220372 12032 raft_consensus.cc:686] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 2 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:52.220499 12032 consensus_queue.cc:237] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 13:59:52.220459 11584 raft_consensus.cc:2932] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:52.221827 6374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f57f1ad5c604b5e957374bab029a5df with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:52.222091 9205 raft_consensus.cc:2932] T 8f57f1ad5c604b5e957374bab029a5df P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:52.223489 6381 catalog_manager.cc:5484] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 1 to 2, leader changed from 0d77752181e944698e809e325e5937e6 (127.0.99.136) to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:52.225863 6790 raft_consensus.cc:2932] T 8f57f1ad5c604b5e957374bab029a5df P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:52.227073 6384 catalog_manager.cc:5484] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index 5 to 6, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:52.229468 6900 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f57f1ad5c604b5e957374bab029a5df with delete_type TABLET_DATA_TOMBSTONED (TS f58065db70b64ef8b1dc499aa6ae4460 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:52.235486 12357 tablet_replica.cc:330] T 8f57f1ad5c604b5e957374bab029a5df P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:52.235829 12357 raft_consensus.cc:2229] T 8f57f1ad5c604b5e957374bab029a5df P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:52.236056 12357 raft_consensus.cc:2258] T 8f57f1ad5c604b5e957374bab029a5df P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:52.236727 12357 ts_tablet_manager.cc:1875] T 8f57f1ad5c604b5e957374bab029a5df P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:52.239775 12357 ts_tablet_manager.cc:1888] T 8f57f1ad5c604b5e957374bab029a5df P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:52.239899 12357 log.cc:1196] T 8f57f1ad5c604b5e957374bab029a5df P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/8f57f1ad5c604b5e957374bab029a5df
I20240503 13:59:52.240468 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet 8f57f1ad5c604b5e957374bab029a5df (table rebalance_test_table_5 [id=cb716c6127694172a7a76a19a23dc99e]) successfully deleted
I20240503 13:59:52.268007 11665 raft_consensus.cc:1050] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735: attempting to promote NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 to VOTER
I20240503 13:59:52.268435 11665 consensus_queue.cc:237] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 13:59:52.269904 8948 raft_consensus.cc:1261] T e2ae8c7089db42199a2e6071800d9efc P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.269920 6528 raft_consensus.cc:1261] T e2ae8c7089db42199a2e6071800d9efc P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.269930 9080 raft_consensus.cc:1261] T e2ae8c7089db42199a2e6071800d9efc P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.270538 12063 consensus_queue.cc:1035] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.270700 12243 consensus_queue.cc:1035] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.270947 11471 consensus_queue.cc:1035] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.272766 12243 raft_consensus.cc:2932] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:52.273223 9080 raft_consensus.cc:2932] T e2ae8c7089db42199a2e6071800d9efc P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:52.273288 6528 raft_consensus.cc:2932] T e2ae8c7089db42199a2e6071800d9efc P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:52.275321 8948 raft_consensus.cc:2932] T e2ae8c7089db42199a2e6071800d9efc P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:52.276451 6381 catalog_manager.cc:5484] T e2ae8c7089db42199a2e6071800d9efc P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:52.278903 12314 raft_consensus.cc:1050] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6: attempting to promote NON_VOTER 2145d15ab2d74c0287839266f1411a43 to VOTER
I20240503 13:59:52.279356 12314 consensus_queue.cc:237] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 13:59:52.281064 6790 raft_consensus.cc:1261] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.281409 9205 consensus_queue.cc:237] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 13:59:52.281571 11690 consensus_queue.cc:1035] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.281854 6660 raft_consensus.cc:1261] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.282330 11690 consensus_queue.cc:1035] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.283241 9080 raft_consensus.cc:1261] T e2ae8c7089db42199a2e6071800d9efc P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.284796 9205 raft_consensus.cc:1261] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.285223 11690 consensus_queue.cc:1035] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.283250 6528 raft_consensus.cc:1261] T e2ae8c7089db42199a2e6071800d9efc P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.285372 12063 consensus_queue.cc:1035] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.285707 11471 consensus_queue.cc:1035] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.286602 11167 raft_consensus.cc:2932] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:52.288046 11471 raft_consensus.cc:2932] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:52.288370 6660 raft_consensus.cc:2932] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:52.289148 6791 raft_consensus.cc:2932] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:52.289183 9205 raft_consensus.cc:2932] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:52.290681 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e2ae8c7089db42199a2e6071800d9efc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:52.291848 6386 catalog_manager.cc:5484] T e2ae8c7089db42199a2e6071800d9efc P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:52.291870 6387 catalog_manager.cc:5484] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:52.292016 6527 raft_consensus.cc:2932] T e2ae8c7089db42199a2e6071800d9efc P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:52.295480 9080 raft_consensus.cc:2932] T e2ae8c7089db42199a2e6071800d9efc P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:52.297978 8926 tablet_service.cc:1514] Processing DeleteTablet for tablet e2ae8c7089db42199a2e6071800d9efc with delete_type TABLET_DATA_TOMBSTONED (TS 06ca8dbc004c42b99faa527a4e6f9545 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:52.298218 12279 tablet_replica.cc:330] T e2ae8c7089db42199a2e6071800d9efc P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:52.298355 12279 raft_consensus.cc:2229] T e2ae8c7089db42199a2e6071800d9efc P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:52.298489 12279 raft_consensus.cc:2258] T e2ae8c7089db42199a2e6071800d9efc P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:52.299625 12279 ts_tablet_manager.cc:1875] T e2ae8c7089db42199a2e6071800d9efc P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:52.301496 12279 ts_tablet_manager.cc:1888] T e2ae8c7089db42199a2e6071800d9efc P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:52.301563 12279 log.cc:1196] T e2ae8c7089db42199a2e6071800d9efc P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/e2ae8c7089db42199a2e6071800d9efc
I20240503 13:59:52.303388 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet e2ae8c7089db42199a2e6071800d9efc (table rebalancer_extra_table_68 [id=2d921adc27d94e8bae2f87cf4caf89d7]) successfully deleted
I20240503 13:59:52.319236 12075 raft_consensus.cc:1050] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545: attempting to promote NON_VOTER 5f46e19644fd4f4b825c86ba6af768da to VOTER
I20240503 13:59:52.319849 12075 consensus_queue.cc:237] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 13:59:52.321466 9343 raft_consensus.cc:1261] T bccac59f680046389bbe23ff262a82d6 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.322137 6527 raft_consensus.cc:1261] T bccac59f680046389bbe23ff262a82d6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.322160 9080 raft_consensus.cc:1261] T bccac59f680046389bbe23ff262a82d6 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.322505 11206 consensus_queue.cc:1035] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.322809 12367 consensus_queue.cc:1035] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.324261 11206 consensus_queue.cc:1035] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.327368 11206 raft_consensus.cc:2932] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:52.327507 9343 raft_consensus.cc:2932] T bccac59f680046389bbe23ff262a82d6 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:52.329633 6527 raft_consensus.cc:2932] T bccac59f680046389bbe23ff262a82d6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:52.331028 9080 raft_consensus.cc:2932] T bccac59f680046389bbe23ff262a82d6 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:52.332126 6384 catalog_manager.cc:5484] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:52.333527 11583 raft_consensus.cc:1050] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4: attempting to promote NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 to VOTER
I20240503 13:59:52.333977 11583 consensus_queue.cc:237] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 13:59:52.336877 6922 raft_consensus.cc:1261] T ee688e32b7d049509a39259d13439cf6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.336999 9205 raft_consensus.cc:1261] T ee688e32b7d049509a39259d13439cf6 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.337321 11371 consensus_queue.cc:1035] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.337448 12180 consensus_queue.cc:1035] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.338060 9343 raft_consensus.cc:1261] T ee688e32b7d049509a39259d13439cf6 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.338466 12180 consensus_queue.cc:1035] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.341413 8948 consensus_queue.cc:237] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 13:59:52.342968 6527 raft_consensus.cc:1261] T bccac59f680046389bbe23ff262a82d6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.343019 9080 raft_consensus.cc:1261] T bccac59f680046389bbe23ff262a82d6 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.343462 12075 consensus_queue.cc:1035] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.343787 12075 consensus_queue.cc:1035] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.344106 11583 raft_consensus.cc:2932] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:52.345487 12367 raft_consensus.cc:2932] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:52.346215 6922 raft_consensus.cc:2932] T ee688e32b7d049509a39259d13439cf6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:52.347080 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bccac59f680046389bbe23ff262a82d6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:52.348086 6527 raft_consensus.cc:2932] T bccac59f680046389bbe23ff262a82d6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:52.348757 6387 catalog_manager.cc:5484] T bccac59f680046389bbe23ff262a82d6 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:52.348951 6381 catalog_manager.cc:5484] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:52.349150 9080 raft_consensus.cc:2932] T bccac59f680046389bbe23ff262a82d6 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:52.350708 9205 raft_consensus.cc:2932] T ee688e32b7d049509a39259d13439cf6 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:52.352661 9321 tablet_service.cc:1514] Processing DeleteTablet for tablet bccac59f680046389bbe23ff262a82d6 with delete_type TABLET_DATA_TOMBSTONED (TS 0d77752181e944698e809e325e5937e6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:52.352861 12350 tablet_replica.cc:330] T bccac59f680046389bbe23ff262a82d6 P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:52.353003 12350 raft_consensus.cc:2229] T bccac59f680046389bbe23ff262a82d6 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:52.353132 12350 raft_consensus.cc:2258] T bccac59f680046389bbe23ff262a82d6 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:52.353729 12350 ts_tablet_manager.cc:1875] T bccac59f680046389bbe23ff262a82d6 P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:52.355942 12350 ts_tablet_manager.cc:1888] T bccac59f680046389bbe23ff262a82d6 P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:52.356014 12350 log.cc:1196] T bccac59f680046389bbe23ff262a82d6 P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/bccac59f680046389bbe23ff262a82d6
I20240503 13:59:52.356973 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet bccac59f680046389bbe23ff262a82d6 (table rebalancer_extra_table_2 [id=e92a6218e0734bbc97220157e127bc15]) successfully deleted
I20240503 13:59:52.357784 9343 raft_consensus.cc:2932] T ee688e32b7d049509a39259d13439cf6 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:52.361877 9080 consensus_queue.cc:237] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 13:59:52.363471 9342 raft_consensus.cc:1261] T ee688e32b7d049509a39259d13439cf6 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.363472 6922 raft_consensus.cc:1261] T ee688e32b7d049509a39259d13439cf6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.363940 12180 consensus_queue.cc:1035] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.364290 12372 consensus_queue.cc:1035] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.365839 11583 raft_consensus.cc:2932] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:52.366225 9342 raft_consensus.cc:2932] T ee688e32b7d049509a39259d13439cf6 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:52.366772 6922 raft_consensus.cc:2932] T ee688e32b7d049509a39259d13439cf6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:52.369753 6387 catalog_manager.cc:5484] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:52.369892 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ee688e32b7d049509a39259d13439cf6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:52.372581 9192 tablet_service.cc:1514] Processing DeleteTablet for tablet ee688e32b7d049509a39259d13439cf6 with delete_type TABLET_DATA_TOMBSTONED (TS ab1d21935d6049888eb0ceccc3c9d735 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57960
I20240503 13:59:52.372817 12339 tablet_replica.cc:330] T ee688e32b7d049509a39259d13439cf6 P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 13:59:52.372956 12339 raft_consensus.cc:2229] T ee688e32b7d049509a39259d13439cf6 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:52.373167 12339 raft_consensus.cc:2258] T ee688e32b7d049509a39259d13439cf6 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:52.373723 12339 ts_tablet_manager.cc:1875] T ee688e32b7d049509a39259d13439cf6 P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:52.375428 12339 ts_tablet_manager.cc:1888] T ee688e32b7d049509a39259d13439cf6 P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:52.375507 12339 log.cc:1196] T ee688e32b7d049509a39259d13439cf6 P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/ee688e32b7d049509a39259d13439cf6
I20240503 13:59:52.375972 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet ee688e32b7d049509a39259d13439cf6 (table rebalancer_extra_table_22 [id=1926a1fe583f4b698c4ba055a9785267]) successfully deleted
I20240503 13:59:52.426955 9342 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8d25b0d68b7643cbbbdeb787cac7a1a4"
dest_uuid: "0d77752181e944698e809e325e5937e6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48440
I20240503 13:59:52.427117 9342 raft_consensus.cc:595] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:52.438011 6922 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "350ba659df804b95b0692091dc2f68a2"
dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33048
I20240503 13:59:52.438186 6922 raft_consensus.cc:595] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:52.513628 11311 raft_consensus.cc:979] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460: : Instructing follower ab1d21935d6049888eb0ceccc3c9d735 to start an election
I20240503 13:59:52.513734 11311 raft_consensus.cc:1067] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Signalling peer ab1d21935d6049888eb0ceccc3c9d735 to start an election
I20240503 13:59:52.514261 9211 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "350ba659df804b95b0692091dc2f68a2"
dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
from {username='slave'} at 127.0.99.132:54795
I20240503 13:59:52.514446 9211 raft_consensus.cc:482] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:52.514520 9211 raft_consensus.cc:3037] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:52.515622 9211 raft_consensus.cc:504] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 13:59:52.516068 9211 leader_election.cc:288] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 election: Requested vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:52.516314 6660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350ba659df804b95b0692091dc2f68a2" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 13:59:52.516441 6660 raft_consensus.cc:3037] T 350ba659df804b95b0692091dc2f68a2 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:52.516425 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350ba659df804b95b0692091dc2f68a2" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:52.516543 6922 raft_consensus.cc:3032] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:52.516588 6922 raft_consensus.cc:729] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 13:59:52.516760 6922 consensus_queue.cc:260] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 13:59:52.516961 6922 raft_consensus.cc:3037] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:52.517205 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350ba659df804b95b0692091dc2f68a2" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:52.517319 6791 raft_consensus.cc:3037] T 350ba659df804b95b0692091dc2f68a2 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:52.517549 6660 raft_consensus.cc:2454] T 350ba659df804b95b0692091dc2f68a2 P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 2.
I20240503 13:59:52.517886 6922 raft_consensus.cc:2454] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 2.
I20240503 13:59:52.518369 6791 raft_consensus.cc:2454] T 350ba659df804b95b0692091dc2f68a2 P 4a85e492c8154e9881a9cc2a105d7e64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 2.
I20240503 13:59:52.518723 9147 leader_election.cc:302] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, ab1d21935d6049888eb0ceccc3c9d735, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:52.518896 12243 raft_consensus.cc:2781] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:52.518983 12243 raft_consensus.cc:686] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 13:59:52.519171 12243 consensus_queue.cc:237] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 13:59:52.521699 6384 catalog_manager.cc:5484] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 1 to 2, leader changed from f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:52.553349 11371 raft_consensus.cc:1050] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4: attempting to promote NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 to VOTER
I20240503 13:59:52.553745 11371 consensus_queue.cc:237] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 13:59:52.555222 6791 raft_consensus.cc:1261] T 62c77ec99cf443d2b7d48b12c8855883 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.555224 9211 raft_consensus.cc:1261] T 62c77ec99cf443d2b7d48b12c8855883 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.555475 6527 raft_consensus.cc:1261] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:52.555617 11583 consensus_queue.cc:1035] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.555770 12372 consensus_queue.cc:1035] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.555943 12180 consensus_queue.cc:1035] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:52.559119 12372 raft_consensus.cc:2932] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:52.559813 6791 raft_consensus.cc:2932] T 62c77ec99cf443d2b7d48b12c8855883 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:52.563194 6384 catalog_manager.cc:5484] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:52.564761 9205 raft_consensus.cc:2932] T 62c77ec99cf443d2b7d48b12c8855883 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:52.567078 6527 raft_consensus.cc:2932] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:52.567436 9080 consensus_queue.cc:237] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 13:59:52.569990 6527 raft_consensus.cc:1261] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.570329 12372 consensus_queue.cc:1035] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.573390 6790 raft_consensus.cc:1261] T 62c77ec99cf443d2b7d48b12c8855883 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.573927 12372 consensus_queue.cc:1035] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.577188 11583 raft_consensus.cc:2932] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:52.577562 6791 raft_consensus.cc:2932] T 62c77ec99cf443d2b7d48b12c8855883 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:52.577580 6527 raft_consensus.cc:2932] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:52.579025 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 62c77ec99cf443d2b7d48b12c8855883 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:52.581383 6380 catalog_manager.cc:5484] T 62c77ec99cf443d2b7d48b12c8855883 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:52.584203 9192 tablet_service.cc:1514] Processing DeleteTablet for tablet 62c77ec99cf443d2b7d48b12c8855883 with delete_type TABLET_DATA_TOMBSTONED (TS ab1d21935d6049888eb0ceccc3c9d735 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57960
I20240503 13:59:52.584424 12339 tablet_replica.cc:330] T 62c77ec99cf443d2b7d48b12c8855883 P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 13:59:52.584564 12339 raft_consensus.cc:2229] T 62c77ec99cf443d2b7d48b12c8855883 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:52.584692 12339 raft_consensus.cc:2258] T 62c77ec99cf443d2b7d48b12c8855883 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:52.584883 9205 consensus_queue.cc:237] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:52.586148 12339 ts_tablet_manager.cc:1875] T 62c77ec99cf443d2b7d48b12c8855883 P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 13:59:52.586577 9148 consensus_peers.cc:485] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): Couldn't send request to peer 2145d15ab2d74c0287839266f1411a43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4cba4ca296e407c9520a6eb7419ed9d. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:52.586794 6527 raft_consensus.cc:1261] T f4cba4ca296e407c9520a6eb7419ed9d P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:52.587155 11665 consensus_queue.cc:1035] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:52.588152 12339 ts_tablet_manager.cc:1888] T 62c77ec99cf443d2b7d48b12c8855883 P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:52.588222 12339 log.cc:1196] T 62c77ec99cf443d2b7d48b12c8855883 P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/62c77ec99cf443d2b7d48b12c8855883
I20240503 13:59:52.588713 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet 62c77ec99cf443d2b7d48b12c8855883 (table rebalancer_extra_table_14 [id=b4dfb7bd2b7f40e9ad8e6e5c28e9ee31]) successfully deleted
I20240503 13:59:52.589023 8948 raft_consensus.cc:1261] T f4cba4ca296e407c9520a6eb7419ed9d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:52.588924 11471 raft_consensus.cc:2932] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:52.589351 6527 raft_consensus.cc:2932] T f4cba4ca296e407c9520a6eb7419ed9d P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:52.589569 12243 consensus_queue.cc:1035] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:52.591123 8948 raft_consensus.cc:2932] T f4cba4ca296e407c9520a6eb7419ed9d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:52.593111 6384 catalog_manager.cc:5484] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:52.605180 6527 consensus_queue.cc:237] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:52.606979 6922 raft_consensus.cc:1261] T f9d875f4872e4f5da12a326381d523c0 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.607035 6791 raft_consensus.cc:1261] T f9d875f4872e4f5da12a326381d523c0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.607457 11894 consensus_queue.cc:1035] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:59:52.607645 6464 consensus_peers.cc:485] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): Couldn't send request to peer 8d574fb4a89d4eea947c859470d44dd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9d875f4872e4f5da12a326381d523c0. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:52.607823 11894 consensus_queue.cc:1035] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.609724 11894 raft_consensus.cc:2932] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:52.610154 6791 raft_consensus.cc:2932] T f9d875f4872e4f5da12a326381d523c0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:52.611097 6922 raft_consensus.cc:2932] T f9d875f4872e4f5da12a326381d523c0 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:52.613941 6380 catalog_manager.cc:5484] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index -1 to 4, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:52.615846 9080 raft_consensus.cc:1261] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 13:59:52.616253 12032 consensus_queue.cc:1035] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.625247 9081 consensus_queue.cc:237] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:52.626721 9342 raft_consensus.cc:1261] T 8482be0fb2784324b77b272e40f57d6b P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:52.626785 8948 raft_consensus.cc:1261] T 8482be0fb2784324b77b272e40f57d6b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:52.627218 12372 consensus_queue.cc:1035] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 13:59:52.627214 9016 consensus_peers.cc:485] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): Couldn't send request to peer 5f46e19644fd4f4b825c86ba6af768da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8482be0fb2784324b77b272e40f57d6b. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:52.627447 11371 consensus_queue.cc:1035] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:52.629360 11583 raft_consensus.cc:2932] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:52.629730 8948 raft_consensus.cc:2932] T 8482be0fb2784324b77b272e40f57d6b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:52.633168 6387 catalog_manager.cc:5484] T 8482be0fb2784324b77b272e40f57d6b P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:52.634778 9342 raft_consensus.cc:2932] T 8482be0fb2784324b77b272e40f57d6b P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:52.642481 9342 consensus_queue.cc:237] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 13:59:52.643867 6922 raft_consensus.cc:1261] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.644197 6790 raft_consensus.cc:1261] T 1350225b1a2848debca68e1bad5528c5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.644589 11378 consensus_queue.cc:1035] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.644769 11690 consensus_queue.cc:1035] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.646425 12314 raft_consensus.cc:2932] T 1350225b1a2848debca68e1bad5528c5 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:52.647106 6921 raft_consensus.cc:2932] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:52.648749 6374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1350225b1a2848debca68e1bad5528c5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:52.650290 8948 consensus_queue.cc:237] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:52.650579 6790 raft_consensus.cc:2932] T 1350225b1a2848debca68e1bad5528c5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:52.651752 6660 raft_consensus.cc:1261] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 13:59:52.652201 12032 consensus_queue.cc:1035] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.652763 9342 raft_consensus.cc:1261] T 98466061b550450bb9d0041427f6f19d P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:52.652973 9081 raft_consensus.cc:1261] T 98466061b550450bb9d0041427f6f19d P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:52.653203 11750 consensus_queue.cc:1035] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:52.654371 11750 consensus_queue.cc:1035] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 13:59:52.655347 8883 consensus_peers.cc:485] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): Couldn't send request to peer ab1d21935d6049888eb0ceccc3c9d735. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98466061b550450bb9d0041427f6f19d. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:52.659160 9342 raft_consensus.cc:1261] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [term 2 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 13:59:52.659575 12181 consensus_queue.cc:1035] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.658613 6381 catalog_manager.cc:5484] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 3 to 4, VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:52.663962 11750 raft_consensus.cc:2932] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:52.665020 6507 tablet_service.cc:1514] Processing DeleteTablet for tablet 1350225b1a2848debca68e1bad5528c5 with delete_type TABLET_DATA_TOMBSTONED (TS 5f46e19644fd4f4b825c86ba6af768da not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:52.669490 6790 consensus_queue.cc:237] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 13:59:52.669684 9342 raft_consensus.cc:2932] T 98466061b550450bb9d0041427f6f19d P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:52.671347 9079 raft_consensus.cc:1261] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:59:52.671720 6660 raft_consensus.cc:1261] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:59:52.671756 12181 consensus_queue.cc:1035] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:52.672051 9081 raft_consensus.cc:2932] T 98466061b550450bb9d0041427f6f19d P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:52.673790 6380 catalog_manager.cc:5484] T 98466061b550450bb9d0041427f6f19d P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index -1 to 2, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:52.674588 12528 raft_consensus.cc:2932] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:52.676064 12181 consensus_queue.cc:1035] T 96ce36eb7c25413eaae380f32b465df7 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:52.677085 6660 raft_consensus.cc:2932] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:52.677300 12532 tablet_replica.cc:330] T 1350225b1a2848debca68e1bad5528c5 P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:52.677454 12532 raft_consensus.cc:2229] T 1350225b1a2848debca68e1bad5528c5 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:52.677685 12532 raft_consensus.cc:2258] T 1350225b1a2848debca68e1bad5528c5 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:52.679420 12532 ts_tablet_manager.cc:1875] T 1350225b1a2848debca68e1bad5528c5 P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:52.680160 9079 raft_consensus.cc:2932] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:52.683341 6381 catalog_manager.cc:5484] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index 3 to 5, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New cstate: current_term: 2 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:52.683725 6372 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 96ce36eb7c25413eaae380f32b465df7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:52.686513 9321 tablet_service.cc:1514] Processing DeleteTablet for tablet 96ce36eb7c25413eaae380f32b465df7 with delete_type TABLET_DATA_TOMBSTONED (TS 0d77752181e944698e809e325e5937e6 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:52.686769 12350 tablet_replica.cc:330] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:52.686901 12350 raft_consensus.cc:2229] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:52.686996 12350 raft_consensus.cc:2258] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:52.687798 12350 ts_tablet_manager.cc:1875] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:52.689777 12350 ts_tablet_manager.cc:1888] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 13:59:52.689846 12350 log.cc:1196] T 96ce36eb7c25413eaae380f32b465df7 P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/96ce36eb7c25413eaae380f32b465df7
I20240503 13:59:52.690277 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet 96ce36eb7c25413eaae380f32b465df7 (table rebalancer_extra_table_46 [id=8e3c65bd2d8c4b5595ccf6d282f48a82]) successfully deleted
I20240503 13:59:52.691717 12532 ts_tablet_manager.cc:1888] T 1350225b1a2848debca68e1bad5528c5 P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:52.691799 12532 log.cc:1196] T 1350225b1a2848debca68e1bad5528c5 P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/1350225b1a2848debca68e1bad5528c5
I20240503 13:59:52.692355 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 1350225b1a2848debca68e1bad5528c5 (table rebalancer_extra_table_58 [id=2f45bc2e41c6493a85eab02853317fd7]) successfully deleted
I20240503 13:59:52.699142 9342 consensus_queue.cc:237] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:52.700488 6790 raft_consensus.cc:1261] T 442c955ab10f4ecbafdc9bf89d8fb39e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:52.700482 6921 raft_consensus.cc:1261] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:52.700611 9279 consensus_peers.cc:485] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 -> Peer 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): Couldn't send request to peer 2145d15ab2d74c0287839266f1411a43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 442c955ab10f4ecbafdc9bf89d8fb39e. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:52.700939 11690 consensus_queue.cc:1035] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:52.701089 12172 consensus_queue.cc:1035] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:52.702867 11167 raft_consensus.cc:2932] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:52.703649 6921 raft_consensus.cc:2932] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:52.703819 6790 raft_consensus.cc:2932] T 442c955ab10f4ecbafdc9bf89d8fb39e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:52.706427 6380 catalog_manager.cc:5484] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:52.714409 9342 consensus_queue.cc:237] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:52.715644 6921 raft_consensus.cc:1261] T 47d38e1d60dd42eabf7191598ded8cd5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:52.715922 9205 raft_consensus.cc:1261] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:52.716022 12314 consensus_queue.cc:1035] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:52.716328 11690 consensus_queue.cc:1035] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 13:59:52.716521 9278 consensus_peers.cc:485] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 -> Peer 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): Couldn't send request to peer 8d574fb4a89d4eea947c859470d44dd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 47d38e1d60dd42eabf7191598ded8cd5. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:52.718034 12314 raft_consensus.cc:2932] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:52.718470 9205 raft_consensus.cc:2932] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:52.718629 6921 raft_consensus.cc:2932] T 47d38e1d60dd42eabf7191598ded8cd5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:52.721690 6381 catalog_manager.cc:5484] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index -1 to 2, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:52.730365 9079 consensus_queue.cc:237] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
W20240503 13:59:52.731856 9017 consensus_peers.cc:485] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ab63490264c4755aeab897f865e9a11. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:52.732192 6921 raft_consensus.cc:1261] T 1ab63490264c4755aeab897f865e9a11 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:52.732292 8948 raft_consensus.cc:1261] T 1ab63490264c4755aeab897f865e9a11 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:52.732698 12180 consensus_queue.cc:1035] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:52.732868 12372 consensus_queue.cc:1035] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:52.734818 11583 raft_consensus.cc:2932] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:52.735711 8948 raft_consensus.cc:2932] T 1ab63490264c4755aeab897f865e9a11 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:52.736207 6921 raft_consensus.cc:2932] T 1ab63490264c4755aeab897f865e9a11 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:52.739650 6387 catalog_manager.cc:5484] T 1ab63490264c4755aeab897f865e9a11 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index -1 to 2, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:52.754439 8948 consensus_queue.cc:237] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:52.755826 9205 raft_consensus.cc:1261] T 3286ea636a5a4d23bf92e8452afd608d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:52.755895 8883 consensus_peers.cc:485] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): Couldn't send request to peer 5f46e19644fd4f4b825c86ba6af768da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3286ea636a5a4d23bf92e8452afd608d. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:52.755901 6660 raft_consensus.cc:1261] T 3286ea636a5a4d23bf92e8452afd608d P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:52.756322 12367 consensus_queue.cc:1035] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:52.756657 11206 consensus_queue.cc:1035] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:52.758387 12075 raft_consensus.cc:2932] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:52.758826 9205 raft_consensus.cc:2932] T 3286ea636a5a4d23bf92e8452afd608d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:52.758826 6660 raft_consensus.cc:2932] T 3286ea636a5a4d23bf92e8452afd608d P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:52.762638 6384 catalog_manager.cc:5484] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:52.770597 8948 consensus_queue.cc:237] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
W20240503 13:59:52.772073 8883 consensus_peers.cc:485] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): Couldn't send request to peer f58065db70b64ef8b1dc499aa6ae4460. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e2d398cb36d54bc785648751b5b52370. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:52.772620 6790 raft_consensus.cc:1261] T e2d398cb36d54bc785648751b5b52370 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:52.772809 6527 raft_consensus.cc:1261] T e2d398cb36d54bc785648751b5b52370 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:52.773017 11206 consensus_queue.cc:1035] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:52.773237 12367 consensus_queue.cc:1035] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:52.775084 11206 raft_consensus.cc:2932] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:52.776719 6527 raft_consensus.cc:2932] T e2d398cb36d54bc785648751b5b52370 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:52.777653 6790 raft_consensus.cc:2932] T e2d398cb36d54bc785648751b5b52370 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:52.781037 6381 catalog_manager.cc:5484] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:52.790758 6527 consensus_queue.cc:237] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:52.792246 6660 raft_consensus.cc:1261] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.792266 6921 raft_consensus.cc:1261] T 39abc23bb3024e699e2f5bdb3bf5a7ea P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:59:52.792326 6464 consensus_peers.cc:485] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da -> Peer 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): Couldn't send request to peer 8d574fb4a89d4eea947c859470d44dd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39abc23bb3024e699e2f5bdb3bf5a7ea. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:52.792640 11894 consensus_queue.cc:1035] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.792817 12508 consensus_queue.cc:1035] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.794525 12508 raft_consensus.cc:2932] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:52.795056 6921 raft_consensus.cc:2932] T 39abc23bb3024e699e2f5bdb3bf5a7ea P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:52.795071 6660 raft_consensus.cc:2932] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:52.798322 6380 catalog_manager.cc:5484] T 39abc23bb3024e699e2f5bdb3bf5a7ea P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index -1 to 4, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:52.807732 6527 consensus_queue.cc:237] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 13:59:52.809378 9342 raft_consensus.cc:1261] T f3dd70554b644cddb1e8eefe7e4ea646 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:52.809782 12508 consensus_queue.cc:1035] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 13:59:52.810263 6464 consensus_peers.cc:485] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): Couldn't send request to peer 06ca8dbc004c42b99faa527a4e6f9545. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3dd70554b644cddb1e8eefe7e4ea646. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:52.810272 6660 raft_consensus.cc:1261] T f3dd70554b644cddb1e8eefe7e4ea646 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:52.810640 12508 consensus_queue.cc:1035] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:52.812065 11894 raft_consensus.cc:2932] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:52.812573 9342 raft_consensus.cc:2932] T f3dd70554b644cddb1e8eefe7e4ea646 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:52.813014 6660 raft_consensus.cc:2932] T f3dd70554b644cddb1e8eefe7e4ea646 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:52.815512 6381 catalog_manager.cc:5484] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index -1 to 2, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:52.826128 11690 raft_consensus.cc:979] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6: : Instructing follower 2145d15ab2d74c0287839266f1411a43 to start an election
I20240503 13:59:52.826237 11690 raft_consensus.cc:1067] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Signalling peer 2145d15ab2d74c0287839266f1411a43 to start an election
I20240503 13:59:52.826728 6660 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8d25b0d68b7643cbbbdeb787cac7a1a4"
dest_uuid: "2145d15ab2d74c0287839266f1411a43"
from {username='slave'} at 127.0.99.136:46439
I20240503 13:59:52.826897 6660 raft_consensus.cc:482] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:52.826957 6660 raft_consensus.cc:3037] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:52.827994 6660 raft_consensus.cc:504] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 13:59:52.828575 6660 leader_election.cc:288] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:52.828878 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d25b0d68b7643cbbbdeb787cac7a1a4" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:52.828995 6790 raft_consensus.cc:3037] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:52.828974 9342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d25b0d68b7643cbbbdeb787cac7a1a4" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:52.829082 9342 raft_consensus.cc:3032] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:52.829118 9342 raft_consensus.cc:729] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Running, Role: LEADER
I20240503 13:59:52.829077 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d25b0d68b7643cbbbdeb787cac7a1a4" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:52.829202 9205 raft_consensus.cc:3037] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:52.829237 9342 consensus_queue.cc:260] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 13:59:52.829388 9342 raft_consensus.cc:3037] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:52.830013 6790 raft_consensus.cc:2454] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 2.
I20240503 13:59:52.830091 9342 raft_consensus.cc:2454] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 2.
I20240503 13:59:52.830145 9205 raft_consensus.cc:2454] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 2.
I20240503 13:59:52.830449 6594 leader_election.cc:302] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 2145d15ab2d74c0287839266f1411a43, 4a85e492c8154e9881a9cc2a105d7e64; no voters:
I20240503 13:59:52.830623 11584 raft_consensus.cc:2781] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:52.830721 11584 raft_consensus.cc:686] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 [term 2 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 13:59:52.830850 11584 consensus_queue.cc:237] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 13:59:52.832485 9342 consensus_queue.cc:237] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:52.833797 6384 catalog_manager.cc:5484] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 1 to 2, leader changed from 0d77752181e944698e809e325e5937e6 (127.0.99.136) to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:52.834093 9205 raft_consensus.cc:1261] T e92eaea6e9c8493690b749f162949ac4 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:52.834231 8948 raft_consensus.cc:1261] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:52.834483 11167 consensus_queue.cc:1035] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:52.834779 11167 consensus_queue.cc:1035] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 13:59:52.837179 9278 consensus_peers.cc:485] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 -> Peer 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): Couldn't send request to peer 5f46e19644fd4f4b825c86ba6af768da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e92eaea6e9c8493690b749f162949ac4. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:52.837607 11690 raft_consensus.cc:2932] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:52.838657 8948 raft_consensus.cc:2932] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:52.839663 9205 raft_consensus.cc:2932] T e92eaea6e9c8493690b749f162949ac4 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:52.841080 6384 catalog_manager.cc:5484] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:52.851156 6790 consensus_queue.cc:237] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:52.852695 9342 raft_consensus.cc:1261] T 64135eb6ef574c95b3677e32d66748a4 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:52.852854 9079 raft_consensus.cc:1261] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:52.853034 12032 consensus_queue.cc:1035] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:52.853220 12181 consensus_queue.cc:1035] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 13:59:52.853592 6726 consensus_peers.cc:485] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 -> Peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): Couldn't send request to peer f58065db70b64ef8b1dc499aa6ae4460. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64135eb6ef574c95b3677e32d66748a4. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:52.855226 12528 raft_consensus.cc:2932] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:52.855876 9342 raft_consensus.cc:2932] T 64135eb6ef574c95b3677e32d66748a4 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:52.858961 6381 catalog_manager.cc:5484] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:52.860064 9079 raft_consensus.cc:2932] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:52.873950 6921 consensus_queue.cc:237] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:52.875265 6527 raft_consensus.cc:1261] T cf7323c2594e4cd5967c3d38e4120815 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.875514 6660 raft_consensus.cc:1261] T cf7323c2594e4cd5967c3d38e4120815 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:52.875666 12266 consensus_queue.cc:1035] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:59:52.876238 6857 consensus_peers.cc:485] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): Couldn't send request to peer 8d574fb4a89d4eea947c859470d44dd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf7323c2594e4cd5967c3d38e4120815. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:52.876340 12266 consensus_queue.cc:1035] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:52.877759 11311 raft_consensus.cc:2932] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:52.878163 6527 raft_consensus.cc:2932] T cf7323c2594e4cd5967c3d38e4120815 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:52.881217 6381 catalog_manager.cc:5484] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index -1 to 4, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:52.882462 6660 raft_consensus.cc:2932] T cf7323c2594e4cd5967c3d38e4120815 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:52.973815 12673 ts_tablet_manager.cc:902] T f4cba4ca296e407c9520a6eb7419ed9d P 2145d15ab2d74c0287839266f1411a43: Initiating tablet copy from peer ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:52.974412 12673 tablet_copy_client.cc:308] T f4cba4ca296e407c9520a6eb7419ed9d P 2145d15ab2d74c0287839266f1411a43: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:34091
I20240503 13:59:52.975371 9231 tablet_copy_service.cc:140] P ab1d21935d6049888eb0ceccc3c9d735: Received BeginTabletCopySession request for tablet f4cba4ca296e407c9520a6eb7419ed9d from peer 2145d15ab2d74c0287839266f1411a43 ({username='slave'} at 127.0.99.130:36297)
I20240503 13:59:52.975555 9231 tablet_copy_service.cc:161] P ab1d21935d6049888eb0ceccc3c9d735: Beginning new tablet copy session on tablet f4cba4ca296e407c9520a6eb7419ed9d from peer 2145d15ab2d74c0287839266f1411a43 at {username='slave'} at 127.0.99.130:36297: session id = 2145d15ab2d74c0287839266f1411a43-f4cba4ca296e407c9520a6eb7419ed9d
I20240503 13:59:52.976562 9231 tablet_copy_source_session.cc:205] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:52.978248 12673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4cba4ca296e407c9520a6eb7419ed9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:52.980914 12673 tablet_copy_client.cc:783] T f4cba4ca296e407c9520a6eb7419ed9d P 2145d15ab2d74c0287839266f1411a43: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:52.981079 12673 tablet_copy_client.cc:647] T f4cba4ca296e407c9520a6eb7419ed9d P 2145d15ab2d74c0287839266f1411a43: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:52.982261 12673 tablet_copy_client.cc:523] T f4cba4ca296e407c9520a6eb7419ed9d P 2145d15ab2d74c0287839266f1411a43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:52.983263 12673 tablet_bootstrap.cc:492] T f4cba4ca296e407c9520a6eb7419ed9d P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:52.986691 12673 tablet_bootstrap.cc:492] T f4cba4ca296e407c9520a6eb7419ed9d P 2145d15ab2d74c0287839266f1411a43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:52.986946 12673 tablet_bootstrap.cc:492] T f4cba4ca296e407c9520a6eb7419ed9d P 2145d15ab2d74c0287839266f1411a43: Bootstrap complete.
I20240503 13:59:52.987115 12673 ts_tablet_manager.cc:1367] T f4cba4ca296e407c9520a6eb7419ed9d P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 13:59:52.987607 12673 raft_consensus.cc:348] T f4cba4ca296e407c9520a6eb7419ed9d P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:52.987756 12673 raft_consensus.cc:729] T f4cba4ca296e407c9520a6eb7419ed9d P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: LEARNER
I20240503 13:59:52.987862 12673 consensus_queue.cc:260] T f4cba4ca296e407c9520a6eb7419ed9d P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:52.989172 12673 ts_tablet_manager.cc:1398] T f4cba4ca296e407c9520a6eb7419ed9d P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:52.990655 9231 tablet_copy_service.cc:342] P ab1d21935d6049888eb0ceccc3c9d735: Request end of tablet copy session 2145d15ab2d74c0287839266f1411a43-f4cba4ca296e407c9520a6eb7419ed9d received from {username='slave'} at 127.0.99.130:36297
I20240503 13:59:52.990741 9231 tablet_copy_service.cc:433] P ab1d21935d6049888eb0ceccc3c9d735: ending tablet copy session 2145d15ab2d74c0287839266f1411a43-f4cba4ca296e407c9520a6eb7419ed9d on tablet f4cba4ca296e407c9520a6eb7419ed9d with peer 2145d15ab2d74c0287839266f1411a43
I20240503 13:59:53.050748 6790 consensus_queue.cc:237] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:53.052081 8948 raft_consensus.cc:1261] T 957b89de328d434d8ea03a59bc93fdcb P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:53.052218 6726 consensus_peers.cc:485] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 -> Peer 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): Couldn't send request to peer 5f46e19644fd4f4b825c86ba6af768da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 957b89de328d434d8ea03a59bc93fdcb. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:53.052418 12032 consensus_queue.cc:1035] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:53.052582 9205 raft_consensus.cc:1261] T 957b89de328d434d8ea03a59bc93fdcb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:53.052920 12032 consensus_queue.cc:1035] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:53.054066 12181 raft_consensus.cc:2932] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:53.054821 8948 raft_consensus.cc:2932] T 957b89de328d434d8ea03a59bc93fdcb P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:53.055434 9205 raft_consensus.cc:2932] T 957b89de328d434d8ea03a59bc93fdcb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:53.057854 6381 catalog_manager.cc:5484] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:53.063112 6660 raft_consensus.cc:1261] T 350ba659df804b95b0692091dc2f68a2 P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 13:59:53.063634 11665 consensus_queue.cc:1035] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:53.068848 9205 consensus_queue.cc:237] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 13:59:53.070031 9342 raft_consensus.cc:1261] T db13f399b9e04fe9a69e729e384d046c P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:53.070283 9147 consensus_peers.cc:485] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): Couldn't send request to peer 06ca8dbc004c42b99faa527a4e6f9545. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db13f399b9e04fe9a69e729e384d046c. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:53.070360 6790 raft_consensus.cc:1261] T db13f399b9e04fe9a69e729e384d046c P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:53.070367 11665 consensus_queue.cc:1035] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:53.070699 11665 consensus_queue.cc:1035] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:53.072268 11665 raft_consensus.cc:2932] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:53.072675 9342 raft_consensus.cc:2932] T db13f399b9e04fe9a69e729e384d046c P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:53.076861 6790 raft_consensus.cc:2932] T db13f399b9e04fe9a69e729e384d046c P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:53.079453 6380 catalog_manager.cc:5484] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index -1 to 2, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:53.084352 6790 consensus_queue.cc:237] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:53.085513 9342 raft_consensus.cc:1261] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:53.085561 8948 raft_consensus.cc:1261] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:53.085666 6726 consensus_peers.cc:485] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 -> Peer 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): Couldn't send request to peer 5f46e19644fd4f4b825c86ba6af768da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c8b2d8a9ed5f4b09812a85445a8e3b85. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:53.085876 12528 consensus_queue.cc:1035] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:53.086192 12528 consensus_queue.cc:1035] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:53.087975 12181 raft_consensus.cc:2932] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:53.088420 8948 raft_consensus.cc:2932] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:53.090902 9342 raft_consensus.cc:2932] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:53.091820 6381 catalog_manager.cc:5484] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:53.095924 6921 raft_consensus.cc:1261] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 13:59:53.096638 12243 consensus_queue.cc:1035] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:53.099148 6790 raft_consensus.cc:1261] T 350ba659df804b95b0692091dc2f68a2 P 4a85e492c8154e9881a9cc2a105d7e64 [term 2 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 13:59:53.099889 11665 consensus_queue.cc:1035] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:53.111423 6921 consensus_queue.cc:237] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:53.112787 9081 raft_consensus.cc:1261] T af8bba02684c4555893def9c1e88ebf0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:53.112934 6660 raft_consensus.cc:1261] T af8bba02684c4555893def9c1e88ebf0 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:53.113162 11271 consensus_queue.cc:1035] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:53.113469 11271 consensus_queue.cc:1035] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 13:59:53.114642 6856 consensus_peers.cc:485] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af8bba02684c4555893def9c1e88ebf0. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:53.115403 12266 raft_consensus.cc:2932] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:53.115832 9081 raft_consensus.cc:2932] T af8bba02684c4555893def9c1e88ebf0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:53.118820 6657 raft_consensus.cc:2932] T af8bba02684c4555893def9c1e88ebf0 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:53.121575 6381 catalog_manager.cc:5484] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:53.131914 12759 ts_tablet_manager.cc:902] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4: Initiating tablet copy from peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:53.136121 12759 tablet_copy_client.cc:308] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:41931
I20240503 13:59:53.136598 9362 tablet_copy_service.cc:140] P 0d77752181e944698e809e325e5937e6: Received BeginTabletCopySession request for tablet 47d38e1d60dd42eabf7191598ded8cd5 from peer 8d574fb4a89d4eea947c859470d44dd4 ({username='slave'} at 127.0.99.134:57883)
I20240503 13:59:53.136710 9362 tablet_copy_service.cc:161] P 0d77752181e944698e809e325e5937e6: Beginning new tablet copy session on tablet 47d38e1d60dd42eabf7191598ded8cd5 from peer 8d574fb4a89d4eea947c859470d44dd4 at {username='slave'} at 127.0.99.134:57883: session id = 8d574fb4a89d4eea947c859470d44dd4-47d38e1d60dd42eabf7191598ded8cd5
I20240503 13:59:53.137728 9362 tablet_copy_source_session.cc:205] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:53.139196 12759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d38e1d60dd42eabf7191598ded8cd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:53.142686 12759 tablet_copy_client.cc:783] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:53.142949 12759 tablet_copy_client.cc:647] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:53.144658 12759 tablet_copy_client.cc:523] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:53.147051 12759 tablet_bootstrap.cc:492] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:53.153491 12759 tablet_bootstrap.cc:492] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:53.153940 12759 tablet_bootstrap.cc:492] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap complete.
I20240503 13:59:53.154106 12759 ts_tablet_manager.cc:1367] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 13:59:53.154477 12775 ts_tablet_manager.cc:902] T 8482be0fb2784324b77b272e40f57d6b P 5f46e19644fd4f4b825c86ba6af768da: Initiating tablet copy from peer 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:53.154584 12776 ts_tablet_manager.cc:902] T f9d875f4872e4f5da12a326381d523c0 P 8d574fb4a89d4eea947c859470d44dd4: Initiating tablet copy from peer 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:53.155246 12776 tablet_copy_client.cc:308] T f9d875f4872e4f5da12a326381d523c0 P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38021
I20240503 13:59:53.154562 12759 raft_consensus.cc:348] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:53.155761 12759 raft_consensus.cc:729] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: LEARNER
I20240503 13:59:53.155680 12775 tablet_copy_client.cc:308] T 8482be0fb2784324b77b272e40f57d6b P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:39481
I20240503 13:59:53.155880 6549 tablet_copy_service.cc:140] P 5f46e19644fd4f4b825c86ba6af768da: Received BeginTabletCopySession request for tablet f9d875f4872e4f5da12a326381d523c0 from peer 8d574fb4a89d4eea947c859470d44dd4 ({username='slave'} at 127.0.99.134:39591)
I20240503 13:59:53.156008 6549 tablet_copy_service.cc:161] P 5f46e19644fd4f4b825c86ba6af768da: Beginning new tablet copy session on tablet f9d875f4872e4f5da12a326381d523c0 from peer 8d574fb4a89d4eea947c859470d44dd4 at {username='slave'} at 127.0.99.134:39591: session id = 8d574fb4a89d4eea947c859470d44dd4-f9d875f4872e4f5da12a326381d523c0
I20240503 13:59:53.155929 12759 consensus_queue.cc:260] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:53.156637 12759 ts_tablet_manager.cc:1398] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:53.157387 6549 tablet_copy_source_session.cc:205] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:53.157533 9362 tablet_copy_service.cc:342] P 0d77752181e944698e809e325e5937e6: Request end of tablet copy session 8d574fb4a89d4eea947c859470d44dd4-47d38e1d60dd42eabf7191598ded8cd5 received from {username='slave'} at 127.0.99.134:57883
I20240503 13:59:53.157624 9362 tablet_copy_service.cc:433] P 0d77752181e944698e809e325e5937e6: ending tablet copy session 8d574fb4a89d4eea947c859470d44dd4-47d38e1d60dd42eabf7191598ded8cd5 on tablet 47d38e1d60dd42eabf7191598ded8cd5 with peer 8d574fb4a89d4eea947c859470d44dd4
I20240503 13:59:53.159583 9101 tablet_copy_service.cc:140] P 8d574fb4a89d4eea947c859470d44dd4: Received BeginTabletCopySession request for tablet 8482be0fb2784324b77b272e40f57d6b from peer 5f46e19644fd4f4b825c86ba6af768da ({username='slave'} at 127.0.99.129:55581)
I20240503 13:59:53.159679 9101 tablet_copy_service.cc:161] P 8d574fb4a89d4eea947c859470d44dd4: Beginning new tablet copy session on tablet 8482be0fb2784324b77b272e40f57d6b from peer 5f46e19644fd4f4b825c86ba6af768da at {username='slave'} at 127.0.99.129:55581: session id = 5f46e19644fd4f4b825c86ba6af768da-8482be0fb2784324b77b272e40f57d6b
I20240503 13:59:53.160543 9101 tablet_copy_source_session.cc:205] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:53.162262 12775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8482be0fb2784324b77b272e40f57d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:53.164554 12775 tablet_copy_client.cc:783] T 8482be0fb2784324b77b272e40f57d6b P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:53.164695 12775 tablet_copy_client.cc:647] T 8482be0fb2784324b77b272e40f57d6b P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:53.167063 12776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9d875f4872e4f5da12a326381d523c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:53.168474 12790 ts_tablet_manager.cc:902] T 3286ea636a5a4d23bf92e8452afd608d P 5f46e19644fd4f4b825c86ba6af768da: Initiating tablet copy from peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:53.169001 12790 tablet_copy_client.cc:308] T 3286ea636a5a4d23bf92e8452afd608d P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:34547
I20240503 13:59:53.169462 8968 tablet_copy_service.cc:140] P 06ca8dbc004c42b99faa527a4e6f9545: Received BeginTabletCopySession request for tablet 3286ea636a5a4d23bf92e8452afd608d from peer 5f46e19644fd4f4b825c86ba6af768da ({username='slave'} at 127.0.99.129:58641)
I20240503 13:59:53.169591 8968 tablet_copy_service.cc:161] P 06ca8dbc004c42b99faa527a4e6f9545: Beginning new tablet copy session on tablet 3286ea636a5a4d23bf92e8452afd608d from peer 5f46e19644fd4f4b825c86ba6af768da at {username='slave'} at 127.0.99.129:58641: session id = 5f46e19644fd4f4b825c86ba6af768da-3286ea636a5a4d23bf92e8452afd608d
I20240503 13:59:53.169865 12775 tablet_copy_client.cc:523] T 8482be0fb2784324b77b272e40f57d6b P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:53.170347 12776 tablet_copy_client.cc:783] T f9d875f4872e4f5da12a326381d523c0 P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:53.170595 12776 tablet_copy_client.cc:647] T f9d875f4872e4f5da12a326381d523c0 P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:53.170645 8968 tablet_copy_source_session.cc:205] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:53.171164 12775 tablet_bootstrap.cc:492] T 8482be0fb2784324b77b272e40f57d6b P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:53.171700 12790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3286ea636a5a4d23bf92e8452afd608d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:53.172344 12776 tablet_copy_client.cc:523] T f9d875f4872e4f5da12a326381d523c0 P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:53.173388 12776 tablet_bootstrap.cc:492] T f9d875f4872e4f5da12a326381d523c0 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:53.175561 12775 tablet_bootstrap.cc:492] T 8482be0fb2784324b77b272e40f57d6b P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:53.175809 12775 tablet_bootstrap.cc:492] T 8482be0fb2784324b77b272e40f57d6b P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap complete.
I20240503 13:59:53.175953 12775 ts_tablet_manager.cc:1367] T 8482be0fb2784324b77b272e40f57d6b P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 13:59:53.176353 12775 raft_consensus.cc:348] T 8482be0fb2784324b77b272e40f57d6b P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:53.176473 12775 raft_consensus.cc:729] T 8482be0fb2784324b77b272e40f57d6b P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: LEARNER
I20240503 13:59:53.176573 12775 consensus_queue.cc:260] T 8482be0fb2784324b77b272e40f57d6b P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:53.176852 12775 ts_tablet_manager.cc:1398] T 8482be0fb2784324b77b272e40f57d6b P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:53.177461 9101 tablet_copy_service.cc:342] P 8d574fb4a89d4eea947c859470d44dd4: Request end of tablet copy session 5f46e19644fd4f4b825c86ba6af768da-8482be0fb2784324b77b272e40f57d6b received from {username='slave'} at 127.0.99.129:55581
I20240503 13:59:53.177536 9101 tablet_copy_service.cc:433] P 8d574fb4a89d4eea947c859470d44dd4: ending tablet copy session 5f46e19644fd4f4b825c86ba6af768da-8482be0fb2784324b77b272e40f57d6b on tablet 8482be0fb2784324b77b272e40f57d6b with peer 5f46e19644fd4f4b825c86ba6af768da
I20240503 13:59:53.178421 12790 tablet_copy_client.cc:783] T 3286ea636a5a4d23bf92e8452afd608d P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:53.178587 12790 tablet_copy_client.cc:647] T 3286ea636a5a4d23bf92e8452afd608d P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:53.184861 12790 tablet_copy_client.cc:523] T 3286ea636a5a4d23bf92e8452afd608d P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:53.186012 12790 tablet_bootstrap.cc:492] T 3286ea636a5a4d23bf92e8452afd608d P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:53.194077 12790 tablet_bootstrap.cc:492] T 3286ea636a5a4d23bf92e8452afd608d P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:53.194552 12790 tablet_bootstrap.cc:492] T 3286ea636a5a4d23bf92e8452afd608d P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap complete.
I20240503 13:59:53.194727 12790 ts_tablet_manager.cc:1367] T 3286ea636a5a4d23bf92e8452afd608d P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240503 13:59:53.195376 12790 raft_consensus.cc:348] T 3286ea636a5a4d23bf92e8452afd608d P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:53.195492 12790 raft_consensus.cc:729] T 3286ea636a5a4d23bf92e8452afd608d P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: LEARNER
I20240503 13:59:53.195592 12790 consensus_queue.cc:260] T 3286ea636a5a4d23bf92e8452afd608d P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:53.196688 12790 ts_tablet_manager.cc:1398] T 3286ea636a5a4d23bf92e8452afd608d P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:53.197808 8968 tablet_copy_service.cc:342] P 06ca8dbc004c42b99faa527a4e6f9545: Request end of tablet copy session 5f46e19644fd4f4b825c86ba6af768da-3286ea636a5a4d23bf92e8452afd608d received from {username='slave'} at 127.0.99.129:58641
I20240503 13:59:53.197886 8968 tablet_copy_service.cc:433] P 06ca8dbc004c42b99faa527a4e6f9545: ending tablet copy session 5f46e19644fd4f4b825c86ba6af768da-3286ea636a5a4d23bf92e8452afd608d on tablet 3286ea636a5a4d23bf92e8452afd608d with peer 5f46e19644fd4f4b825c86ba6af768da
I20240503 13:59:53.205550 12813 ts_tablet_manager.cc:902] T 98466061b550450bb9d0041427f6f19d P ab1d21935d6049888eb0ceccc3c9d735: Initiating tablet copy from peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:53.205597 12776 tablet_bootstrap.cc:492] T f9d875f4872e4f5da12a326381d523c0 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:53.205890 12776 tablet_bootstrap.cc:492] T f9d875f4872e4f5da12a326381d523c0 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap complete.
I20240503 13:59:53.206040 12813 tablet_copy_client.cc:308] T 98466061b550450bb9d0041427f6f19d P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:34547
I20240503 13:59:53.206112 12776 ts_tablet_manager.cc:1367] T f9d875f4872e4f5da12a326381d523c0 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.033s user 0.028s sys 0.000s
I20240503 13:59:53.206553 12776 raft_consensus.cc:348] T f9d875f4872e4f5da12a326381d523c0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:53.206749 12776 raft_consensus.cc:729] T f9d875f4872e4f5da12a326381d523c0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: LEARNER
I20240503 13:59:53.206749 8968 tablet_copy_service.cc:140] P 06ca8dbc004c42b99faa527a4e6f9545: Received BeginTabletCopySession request for tablet 98466061b550450bb9d0041427f6f19d from peer ab1d21935d6049888eb0ceccc3c9d735 ({username='slave'} at 127.0.99.135:33143)
I20240503 13:59:53.206842 8968 tablet_copy_service.cc:161] P 06ca8dbc004c42b99faa527a4e6f9545: Beginning new tablet copy session on tablet 98466061b550450bb9d0041427f6f19d from peer ab1d21935d6049888eb0ceccc3c9d735 at {username='slave'} at 127.0.99.135:33143: session id = ab1d21935d6049888eb0ceccc3c9d735-98466061b550450bb9d0041427f6f19d
I20240503 13:59:53.206918 12776 consensus_queue.cc:260] T f9d875f4872e4f5da12a326381d523c0 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:53.207319 12776 ts_tablet_manager.cc:1398] T f9d875f4872e4f5da12a326381d523c0 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:53.207845 8968 tablet_copy_source_session.cc:205] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:53.208276 6549 tablet_copy_service.cc:342] P 5f46e19644fd4f4b825c86ba6af768da: Request end of tablet copy session 8d574fb4a89d4eea947c859470d44dd4-f9d875f4872e4f5da12a326381d523c0 received from {username='slave'} at 127.0.99.134:39591
I20240503 13:59:53.208354 6549 tablet_copy_service.cc:433] P 5f46e19644fd4f4b825c86ba6af768da: ending tablet copy session 8d574fb4a89d4eea947c859470d44dd4-f9d875f4872e4f5da12a326381d523c0 on tablet f9d875f4872e4f5da12a326381d523c0 with peer 8d574fb4a89d4eea947c859470d44dd4
I20240503 13:59:53.208700 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98466061b550450bb9d0041427f6f19d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:53.210785 12813 tablet_copy_client.cc:783] T 98466061b550450bb9d0041427f6f19d P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:53.210927 12813 tablet_copy_client.cc:647] T 98466061b550450bb9d0041427f6f19d P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:53.212708 12813 tablet_copy_client.cc:523] T 98466061b550450bb9d0041427f6f19d P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:53.213961 12813 tablet_bootstrap.cc:492] T 98466061b550450bb9d0041427f6f19d P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:53.217278 12813 tablet_bootstrap.cc:492] T 98466061b550450bb9d0041427f6f19d P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:53.217540 12813 tablet_bootstrap.cc:492] T 98466061b550450bb9d0041427f6f19d P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap complete.
I20240503 13:59:53.217830 12813 ts_tablet_manager.cc:1367] T 98466061b550450bb9d0041427f6f19d P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 13:59:53.218294 12813 raft_consensus.cc:348] T 98466061b550450bb9d0041427f6f19d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:53.218444 12813 raft_consensus.cc:729] T 98466061b550450bb9d0041427f6f19d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: LEARNER
I20240503 13:59:53.218684 12813 consensus_queue.cc:260] T 98466061b550450bb9d0041427f6f19d P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:53.219038 12813 ts_tablet_manager.cc:1398] T 98466061b550450bb9d0041427f6f19d P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:53.220687 8968 tablet_copy_service.cc:342] P 06ca8dbc004c42b99faa527a4e6f9545: Request end of tablet copy session ab1d21935d6049888eb0ceccc3c9d735-98466061b550450bb9d0041427f6f19d received from {username='slave'} at 127.0.99.135:33143
I20240503 13:59:53.220765 8968 tablet_copy_service.cc:433] P 06ca8dbc004c42b99faa527a4e6f9545: ending tablet copy session ab1d21935d6049888eb0ceccc3c9d735-98466061b550450bb9d0041427f6f19d on tablet 98466061b550450bb9d0041427f6f19d with peer ab1d21935d6049888eb0ceccc3c9d735
I20240503 13:59:53.223723 9211 raft_consensus.cc:1261] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 13:59:53.224315 11584 consensus_queue.cc:1035] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:53.231237 12673 ts_tablet_manager.cc:902] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43: Initiating tablet copy from peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:53.231664 12673 tablet_copy_client.cc:308] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:41931
I20240503 13:59:53.232115 9362 tablet_copy_service.cc:140] P 0d77752181e944698e809e325e5937e6: Received BeginTabletCopySession request for tablet 442c955ab10f4ecbafdc9bf89d8fb39e from peer 2145d15ab2d74c0287839266f1411a43 ({username='slave'} at 127.0.99.130:49785)
I20240503 13:59:53.232216 9362 tablet_copy_service.cc:161] P 0d77752181e944698e809e325e5937e6: Beginning new tablet copy session on tablet 442c955ab10f4ecbafdc9bf89d8fb39e from peer 2145d15ab2d74c0287839266f1411a43 at {username='slave'} at 127.0.99.130:49785: session id = 2145d15ab2d74c0287839266f1411a43-442c955ab10f4ecbafdc9bf89d8fb39e
I20240503 13:59:53.233163 9362 tablet_copy_source_session.cc:205] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:53.234314 12673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442c955ab10f4ecbafdc9bf89d8fb39e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:53.237121 12673 tablet_copy_client.cc:783] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:53.237295 12673 tablet_copy_client.cc:647] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:53.239337 12673 tablet_copy_client.cc:523] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:53.240830 12673 tablet_bootstrap.cc:492] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:53.244305 12673 tablet_bootstrap.cc:492] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:53.244495 12673 tablet_bootstrap.cc:492] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43: Bootstrap complete.
I20240503 13:59:53.244601 12673 ts_tablet_manager.cc:1367] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 13:59:53.245009 12673 raft_consensus.cc:348] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:53.245150 12673 raft_consensus.cc:729] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: LEARNER
I20240503 13:59:53.245293 12673 consensus_queue.cc:260] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:53.245638 12673 ts_tablet_manager.cc:1398] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 13:59:53.247556 9362 tablet_copy_service.cc:342] P 0d77752181e944698e809e325e5937e6: Request end of tablet copy session 2145d15ab2d74c0287839266f1411a43-442c955ab10f4ecbafdc9bf89d8fb39e received from {username='slave'} at 127.0.99.130:49785
I20240503 13:59:53.247648 9362 tablet_copy_service.cc:433] P 0d77752181e944698e809e325e5937e6: ending tablet copy session 2145d15ab2d74c0287839266f1411a43-442c955ab10f4ecbafdc9bf89d8fb39e on tablet 442c955ab10f4ecbafdc9bf89d8fb39e with peer 2145d15ab2d74c0287839266f1411a43
I20240503 13:59:53.280620 12776 ts_tablet_manager.cc:902] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 8d574fb4a89d4eea947c859470d44dd4: Initiating tablet copy from peer 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:53.281119 12776 tablet_copy_client.cc:308] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38021
I20240503 13:59:53.281996 6549 tablet_copy_service.cc:140] P 5f46e19644fd4f4b825c86ba6af768da: Received BeginTabletCopySession request for tablet 39abc23bb3024e699e2f5bdb3bf5a7ea from peer 8d574fb4a89d4eea947c859470d44dd4 ({username='slave'} at 127.0.99.134:39591)
I20240503 13:59:53.282125 6549 tablet_copy_service.cc:161] P 5f46e19644fd4f4b825c86ba6af768da: Beginning new tablet copy session on tablet 39abc23bb3024e699e2f5bdb3bf5a7ea from peer 8d574fb4a89d4eea947c859470d44dd4 at {username='slave'} at 127.0.99.134:39591: session id = 8d574fb4a89d4eea947c859470d44dd4-39abc23bb3024e699e2f5bdb3bf5a7ea
I20240503 13:59:53.283008 6549 tablet_copy_source_session.cc:205] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:53.283823 12776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39abc23bb3024e699e2f5bdb3bf5a7ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:53.286614 12776 tablet_copy_client.cc:783] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:53.286784 12776 tablet_copy_client.cc:647] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:53.287777 12776 tablet_copy_client.cc:523] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:53.288851 12776 tablet_bootstrap.cc:492] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:53.295356 12885 ts_tablet_manager.cc:902] T f3dd70554b644cddb1e8eefe7e4ea646 P 06ca8dbc004c42b99faa527a4e6f9545: Initiating tablet copy from peer 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:53.296033 12885 tablet_copy_client.cc:308] T f3dd70554b644cddb1e8eefe7e4ea646 P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38021
I20240503 13:59:53.296603 6549 tablet_copy_service.cc:140] P 5f46e19644fd4f4b825c86ba6af768da: Received BeginTabletCopySession request for tablet f3dd70554b644cddb1e8eefe7e4ea646 from peer 06ca8dbc004c42b99faa527a4e6f9545 ({username='slave'} at 127.0.99.133:56195)
I20240503 13:59:53.296710 6549 tablet_copy_service.cc:161] P 5f46e19644fd4f4b825c86ba6af768da: Beginning new tablet copy session on tablet f3dd70554b644cddb1e8eefe7e4ea646 from peer 06ca8dbc004c42b99faa527a4e6f9545 at {username='slave'} at 127.0.99.133:56195: session id = 06ca8dbc004c42b99faa527a4e6f9545-f3dd70554b644cddb1e8eefe7e4ea646
I20240503 13:59:53.297741 6549 tablet_copy_source_session.cc:205] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:53.298731 12885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3dd70554b644cddb1e8eefe7e4ea646. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:53.302659 12885 tablet_copy_client.cc:783] T f3dd70554b644cddb1e8eefe7e4ea646 P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:53.302912 12885 tablet_copy_client.cc:647] T f3dd70554b644cddb1e8eefe7e4ea646 P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:53.308557 12885 tablet_copy_client.cc:523] T f3dd70554b644cddb1e8eefe7e4ea646 P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:53.310127 12885 tablet_bootstrap.cc:492] T f3dd70554b644cddb1e8eefe7e4ea646 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:53.322422 12776 tablet_bootstrap.cc:492] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:53.322692 12776 tablet_bootstrap.cc:492] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap complete.
I20240503 13:59:53.322860 12776 ts_tablet_manager.cc:1367] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.034s user 0.024s sys 0.000s
I20240503 13:59:53.323254 12776 raft_consensus.cc:348] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:53.323457 12776 raft_consensus.cc:729] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: LEARNER
I20240503 13:59:53.323585 12776 consensus_queue.cc:260] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:53.323972 12776 ts_tablet_manager.cc:1398] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 13:59:53.324540 12885 tablet_bootstrap.cc:492] T f3dd70554b644cddb1e8eefe7e4ea646 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:53.324873 12885 tablet_bootstrap.cc:492] T f3dd70554b644cddb1e8eefe7e4ea646 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap complete.
I20240503 13:59:53.325037 12885 ts_tablet_manager.cc:1367] T f3dd70554b644cddb1e8eefe7e4ea646 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240503 13:59:53.325560 12885 raft_consensus.cc:348] T f3dd70554b644cddb1e8eefe7e4ea646 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 13:59:53.325683 12885 raft_consensus.cc:729] T f3dd70554b644cddb1e8eefe7e4ea646 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: LEARNER
I20240503 13:59:53.325789 12885 consensus_queue.cc:260] T f3dd70554b644cddb1e8eefe7e4ea646 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 13:59:53.326048 12885 ts_tablet_manager.cc:1398] T f3dd70554b644cddb1e8eefe7e4ea646 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:53.326838 6549 tablet_copy_service.cc:342] P 5f46e19644fd4f4b825c86ba6af768da: Request end of tablet copy session 8d574fb4a89d4eea947c859470d44dd4-39abc23bb3024e699e2f5bdb3bf5a7ea received from {username='slave'} at 127.0.99.134:39591
I20240503 13:59:53.326915 6549 tablet_copy_service.cc:433] P 5f46e19644fd4f4b825c86ba6af768da: ending tablet copy session 8d574fb4a89d4eea947c859470d44dd4-39abc23bb3024e699e2f5bdb3bf5a7ea on tablet 39abc23bb3024e699e2f5bdb3bf5a7ea with peer 8d574fb4a89d4eea947c859470d44dd4
I20240503 13:59:53.327759 6549 tablet_copy_service.cc:342] P 5f46e19644fd4f4b825c86ba6af768da: Request end of tablet copy session 06ca8dbc004c42b99faa527a4e6f9545-f3dd70554b644cddb1e8eefe7e4ea646 received from {username='slave'} at 127.0.99.133:56195
I20240503 13:59:53.327826 6549 tablet_copy_service.cc:433] P 5f46e19644fd4f4b825c86ba6af768da: ending tablet copy session 06ca8dbc004c42b99faa527a4e6f9545-f3dd70554b644cddb1e8eefe7e4ea646 on tablet f3dd70554b644cddb1e8eefe7e4ea646 with peer 06ca8dbc004c42b99faa527a4e6f9545
I20240503 13:59:53.327999 12896 ts_tablet_manager.cc:902] T e2d398cb36d54bc785648751b5b52370 P f58065db70b64ef8b1dc499aa6ae4460: Initiating tablet copy from peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:53.328609 12896 tablet_copy_client.cc:308] T e2d398cb36d54bc785648751b5b52370 P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:34547
I20240503 13:59:53.330202 8968 tablet_copy_service.cc:140] P 06ca8dbc004c42b99faa527a4e6f9545: Received BeginTabletCopySession request for tablet e2d398cb36d54bc785648751b5b52370 from peer f58065db70b64ef8b1dc499aa6ae4460 ({username='slave'} at 127.0.99.132:34657)
I20240503 13:59:53.330303 8968 tablet_copy_service.cc:161] P 06ca8dbc004c42b99faa527a4e6f9545: Beginning new tablet copy session on tablet e2d398cb36d54bc785648751b5b52370 from peer f58065db70b64ef8b1dc499aa6ae4460 at {username='slave'} at 127.0.99.132:34657: session id = f58065db70b64ef8b1dc499aa6ae4460-e2d398cb36d54bc785648751b5b52370
I20240503 13:59:53.331310 8968 tablet_copy_source_session.cc:205] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:53.332288 12896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d398cb36d54bc785648751b5b52370. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:53.335981 12896 tablet_copy_client.cc:783] T e2d398cb36d54bc785648751b5b52370 P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:53.336129 12896 tablet_copy_client.cc:647] T e2d398cb36d54bc785648751b5b52370 P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:53.337080 12896 tablet_copy_client.cc:523] T e2d398cb36d54bc785648751b5b52370 P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:53.338241 12896 tablet_bootstrap.cc:492] T e2d398cb36d54bc785648751b5b52370 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:53.341418 12896 tablet_bootstrap.cc:492] T e2d398cb36d54bc785648751b5b52370 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:53.341631 12896 tablet_bootstrap.cc:492] T e2d398cb36d54bc785648751b5b52370 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap complete.
I20240503 13:59:53.341773 12896 ts_tablet_manager.cc:1367] T e2d398cb36d54bc785648751b5b52370 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 13:59:53.342113 12896 raft_consensus.cc:348] T e2d398cb36d54bc785648751b5b52370 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:53.342209 12896 raft_consensus.cc:729] T e2d398cb36d54bc785648751b5b52370 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: LEARNER
I20240503 13:59:53.342330 12896 consensus_queue.cc:260] T e2d398cb36d54bc785648751b5b52370 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:53.342556 12896 ts_tablet_manager.cc:1398] T e2d398cb36d54bc785648751b5b52370 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 13:59:53.343029 8968 tablet_copy_service.cc:342] P 06ca8dbc004c42b99faa527a4e6f9545: Request end of tablet copy session f58065db70b64ef8b1dc499aa6ae4460-e2d398cb36d54bc785648751b5b52370 received from {username='slave'} at 127.0.99.132:34657
I20240503 13:59:53.343097 8968 tablet_copy_service.cc:433] P 06ca8dbc004c42b99faa527a4e6f9545: ending tablet copy session f58065db70b64ef8b1dc499aa6ae4460-e2d398cb36d54bc785648751b5b52370 on tablet e2d398cb36d54bc785648751b5b52370 with peer f58065db70b64ef8b1dc499aa6ae4460
I20240503 13:59:53.352042 12902 ts_tablet_manager.cc:902] T 1ab63490264c4755aeab897f865e9a11 P 0d77752181e944698e809e325e5937e6: Initiating tablet copy from peer 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:53.353058 12902 tablet_copy_client.cc:308] T 1ab63490264c4755aeab897f865e9a11 P 0d77752181e944698e809e325e5937e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:39481
I20240503 13:59:53.353493 9101 tablet_copy_service.cc:140] P 8d574fb4a89d4eea947c859470d44dd4: Received BeginTabletCopySession request for tablet 1ab63490264c4755aeab897f865e9a11 from peer 0d77752181e944698e809e325e5937e6 ({username='slave'} at 127.0.99.136:49573)
I20240503 13:59:53.353590 9101 tablet_copy_service.cc:161] P 8d574fb4a89d4eea947c859470d44dd4: Beginning new tablet copy session on tablet 1ab63490264c4755aeab897f865e9a11 from peer 0d77752181e944698e809e325e5937e6 at {username='slave'} at 127.0.99.136:49573: session id = 0d77752181e944698e809e325e5937e6-1ab63490264c4755aeab897f865e9a11
I20240503 13:59:53.354458 9101 tablet_copy_source_session.cc:205] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:53.355209 12902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab63490264c4755aeab897f865e9a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:53.356940 12902 tablet_copy_client.cc:783] T 1ab63490264c4755aeab897f865e9a11 P 0d77752181e944698e809e325e5937e6: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:53.357095 12902 tablet_copy_client.cc:647] T 1ab63490264c4755aeab897f865e9a11 P 0d77752181e944698e809e325e5937e6: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:53.358047 12902 tablet_copy_client.cc:523] T 1ab63490264c4755aeab897f865e9a11 P 0d77752181e944698e809e325e5937e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:53.358959 12902 tablet_bootstrap.cc:492] T 1ab63490264c4755aeab897f865e9a11 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:53.362258 12902 tablet_bootstrap.cc:492] T 1ab63490264c4755aeab897f865e9a11 P 0d77752181e944698e809e325e5937e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:53.362473 12902 tablet_bootstrap.cc:492] T 1ab63490264c4755aeab897f865e9a11 P 0d77752181e944698e809e325e5937e6: Bootstrap complete.
I20240503 13:59:53.362623 12902 ts_tablet_manager.cc:1367] T 1ab63490264c4755aeab897f865e9a11 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 13:59:53.362955 12902 raft_consensus.cc:348] T 1ab63490264c4755aeab897f865e9a11 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:53.363049 12902 raft_consensus.cc:729] T 1ab63490264c4755aeab897f865e9a11 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: LEARNER
I20240503 13:59:53.363124 12902 consensus_queue.cc:260] T 1ab63490264c4755aeab897f865e9a11 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:53.363457 12902 ts_tablet_manager.cc:1398] T 1ab63490264c4755aeab897f865e9a11 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:53.363960 9101 tablet_copy_service.cc:342] P 8d574fb4a89d4eea947c859470d44dd4: Request end of tablet copy session 0d77752181e944698e809e325e5937e6-1ab63490264c4755aeab897f865e9a11 received from {username='slave'} at 127.0.99.136:49573
I20240503 13:59:53.364020 9101 tablet_copy_service.cc:433] P 8d574fb4a89d4eea947c859470d44dd4: ending tablet copy session 0d77752181e944698e809e325e5937e6-1ab63490264c4755aeab897f865e9a11 on tablet 1ab63490264c4755aeab897f865e9a11 with peer 0d77752181e944698e809e325e5937e6
I20240503 13:59:53.416555 12896 ts_tablet_manager.cc:902] T 64135eb6ef574c95b3677e32d66748a4 P f58065db70b64ef8b1dc499aa6ae4460: Initiating tablet copy from peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:53.417032 12896 tablet_copy_client.cc:308] T 64135eb6ef574c95b3677e32d66748a4 P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:46365
I20240503 13:59:53.417585 6811 tablet_copy_service.cc:140] P 4a85e492c8154e9881a9cc2a105d7e64: Received BeginTabletCopySession request for tablet 64135eb6ef574c95b3677e32d66748a4 from peer f58065db70b64ef8b1dc499aa6ae4460 ({username='slave'} at 127.0.99.132:42563)
I20240503 13:59:53.417713 6811 tablet_copy_service.cc:161] P 4a85e492c8154e9881a9cc2a105d7e64: Beginning new tablet copy session on tablet 64135eb6ef574c95b3677e32d66748a4 from peer f58065db70b64ef8b1dc499aa6ae4460 at {username='slave'} at 127.0.99.132:42563: session id = f58065db70b64ef8b1dc499aa6ae4460-64135eb6ef574c95b3677e32d66748a4
I20240503 13:59:53.419036 6811 tablet_copy_source_session.cc:205] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:53.420058 12896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64135eb6ef574c95b3677e32d66748a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:53.421697 12896 tablet_copy_client.cc:783] T 64135eb6ef574c95b3677e32d66748a4 P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:53.421844 12896 tablet_copy_client.cc:647] T 64135eb6ef574c95b3677e32d66748a4 P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:53.423072 12896 tablet_copy_client.cc:523] T 64135eb6ef574c95b3677e32d66748a4 P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:53.424049 12896 tablet_bootstrap.cc:492] T 64135eb6ef574c95b3677e32d66748a4 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:53.426929 12896 tablet_bootstrap.cc:492] T 64135eb6ef574c95b3677e32d66748a4 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:53.427165 12896 tablet_bootstrap.cc:492] T 64135eb6ef574c95b3677e32d66748a4 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap complete.
I20240503 13:59:53.427332 12896 ts_tablet_manager.cc:1367] T 64135eb6ef574c95b3677e32d66748a4 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240503 13:59:53.427783 12896 raft_consensus.cc:348] T 64135eb6ef574c95b3677e32d66748a4 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:53.427877 12896 raft_consensus.cc:729] T 64135eb6ef574c95b3677e32d66748a4 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: LEARNER
I20240503 13:59:53.427954 12896 consensus_queue.cc:260] T 64135eb6ef574c95b3677e32d66748a4 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:53.428275 12896 ts_tablet_manager.cc:1398] T 64135eb6ef574c95b3677e32d66748a4 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:53.428898 6811 tablet_copy_service.cc:342] P 4a85e492c8154e9881a9cc2a105d7e64: Request end of tablet copy session f58065db70b64ef8b1dc499aa6ae4460-64135eb6ef574c95b3677e32d66748a4 received from {username='slave'} at 127.0.99.132:42563
I20240503 13:59:53.428977 6811 tablet_copy_service.cc:433] P 4a85e492c8154e9881a9cc2a105d7e64: ending tablet copy session f58065db70b64ef8b1dc499aa6ae4460-64135eb6ef574c95b3677e32d66748a4 on tablet 64135eb6ef574c95b3677e32d66748a4 with peer f58065db70b64ef8b1dc499aa6ae4460
I20240503 13:59:53.447088 6790 raft_consensus.cc:1261] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 2 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 13:59:53.447656 11584 consensus_queue.cc:1035] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:53.451987 12790 ts_tablet_manager.cc:902] T e92eaea6e9c8493690b749f162949ac4 P 5f46e19644fd4f4b825c86ba6af768da: Initiating tablet copy from peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:53.452253 9342 raft_consensus.cc:1261] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [term 2 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 13:59:53.452461 12790 tablet_copy_client.cc:308] T e92eaea6e9c8493690b749f162949ac4 P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:41931
I20240503 13:59:53.452797 11584 consensus_queue.cc:1035] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:53.453249 9362 tablet_copy_service.cc:140] P 0d77752181e944698e809e325e5937e6: Received BeginTabletCopySession request for tablet e92eaea6e9c8493690b749f162949ac4 from peer 5f46e19644fd4f4b825c86ba6af768da ({username='slave'} at 127.0.99.129:55285)
I20240503 13:59:53.453346 9362 tablet_copy_service.cc:161] P 0d77752181e944698e809e325e5937e6: Beginning new tablet copy session on tablet e92eaea6e9c8493690b749f162949ac4 from peer 5f46e19644fd4f4b825c86ba6af768da at {username='slave'} at 127.0.99.129:55285: session id = 5f46e19644fd4f4b825c86ba6af768da-e92eaea6e9c8493690b749f162949ac4
I20240503 13:59:53.454010 6657 consensus_queue.cc:237] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 13:59:53.454260 9362 tablet_copy_source_session.cc:205] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:53.455197 12790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e92eaea6e9c8493690b749f162949ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:53.455854 9211 raft_consensus.cc:1261] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:59:53.457449 6788 raft_consensus.cc:1261] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 2 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:59:53.457681 12790 tablet_copy_client.cc:783] T e92eaea6e9c8493690b749f162949ac4 P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:53.457722 11584 consensus_queue.cc:1035] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:53.457859 12790 tablet_copy_client.cc:647] T e92eaea6e9c8493690b749f162949ac4 P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:53.458138 11584 consensus_queue.cc:1035] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:53.460929 12914 raft_consensus.cc:2932] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:53.461325 9211 raft_consensus.cc:2932] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:53.463991 6374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8d25b0d68b7643cbbbdeb787cac7a1a4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:53.465013 6381 catalog_manager.cc:5484] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index 3 to 5, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New cstate: current_term: 2 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:53.465924 12776 ts_tablet_manager.cc:902] T cf7323c2594e4cd5967c3d38e4120815 P 8d574fb4a89d4eea947c859470d44dd4: Initiating tablet copy from peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:53.466444 12776 tablet_copy_client.cc:308] T cf7323c2594e4cd5967c3d38e4120815 P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43531
I20240503 13:59:53.466930 6942 tablet_copy_service.cc:140] P f58065db70b64ef8b1dc499aa6ae4460: Received BeginTabletCopySession request for tablet cf7323c2594e4cd5967c3d38e4120815 from peer 8d574fb4a89d4eea947c859470d44dd4 ({username='slave'} at 127.0.99.134:57297)
I20240503 13:59:53.467027 6942 tablet_copy_service.cc:161] P f58065db70b64ef8b1dc499aa6ae4460: Beginning new tablet copy session on tablet cf7323c2594e4cd5967c3d38e4120815 from peer 8d574fb4a89d4eea947c859470d44dd4 at {username='slave'} at 127.0.99.134:57297: session id = 8d574fb4a89d4eea947c859470d44dd4-cf7323c2594e4cd5967c3d38e4120815
I20240503 13:59:53.467895 6942 tablet_copy_source_session.cc:205] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:53.468465 9321 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d25b0d68b7643cbbbdeb787cac7a1a4 with delete_type TABLET_DATA_TOMBSTONED (TS 0d77752181e944698e809e325e5937e6 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:53.469465 12776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7323c2594e4cd5967c3d38e4120815. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:53.469468 6657 raft_consensus.cc:1203] T f4cba4ca296e407c9520a6eb7419ed9d P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:53.471491 12776 tablet_copy_client.cc:783] T cf7323c2594e4cd5967c3d38e4120815 P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:53.471639 12776 tablet_copy_client.cc:647] T cf7323c2594e4cd5967c3d38e4120815 P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:53.473124 12776 tablet_copy_client.cc:523] T cf7323c2594e4cd5967c3d38e4120815 P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:53.473362 6790 raft_consensus.cc:2932] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:53.474452 12776 tablet_bootstrap.cc:492] T cf7323c2594e4cd5967c3d38e4120815 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:53.476728 12790 tablet_copy_client.cc:523] T e92eaea6e9c8493690b749f162949ac4 P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:53.476926 12926 tablet_replica.cc:330] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:53.477102 12926 raft_consensus.cc:2229] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:53.477236 12926 raft_consensus.cc:2258] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:53.477784 12926 ts_tablet_manager.cc:1875] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:53.478113 12790 tablet_bootstrap.cc:492] T e92eaea6e9c8493690b749f162949ac4 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:53.480072 12926 ts_tablet_manager.cc:1888] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 13:59:53.480190 12926 log.cc:1196] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/8d25b0d68b7643cbbbdeb787cac7a1a4
I20240503 13:59:53.480796 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet 8d25b0d68b7643cbbbdeb787cac7a1a4 (table rebalancer_extra_table_8 [id=9e8aad8638e34703aac55865282ca799]) successfully deleted
I20240503 13:59:53.483999 12790 tablet_bootstrap.cc:492] T e92eaea6e9c8493690b749f162949ac4 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:53.484247 12790 tablet_bootstrap.cc:492] T e92eaea6e9c8493690b749f162949ac4 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap complete.
I20240503 13:59:53.484377 12790 ts_tablet_manager.cc:1367] T e92eaea6e9c8493690b749f162949ac4 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 13:59:53.484714 12790 raft_consensus.cc:348] T e92eaea6e9c8493690b749f162949ac4 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:53.484808 12790 raft_consensus.cc:729] T e92eaea6e9c8493690b749f162949ac4 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: LEARNER
I20240503 13:59:53.484880 12790 consensus_queue.cc:260] T e92eaea6e9c8493690b749f162949ac4 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:53.485137 12790 ts_tablet_manager.cc:1398] T e92eaea6e9c8493690b749f162949ac4 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:53.486196 9362 tablet_copy_service.cc:342] P 0d77752181e944698e809e325e5937e6: Request end of tablet copy session 5f46e19644fd4f4b825c86ba6af768da-e92eaea6e9c8493690b749f162949ac4 received from {username='slave'} at 127.0.99.129:55285
I20240503 13:59:53.486263 9362 tablet_copy_service.cc:433] P 0d77752181e944698e809e325e5937e6: ending tablet copy session 5f46e19644fd4f4b825c86ba6af768da-e92eaea6e9c8493690b749f162949ac4 on tablet e92eaea6e9c8493690b749f162949ac4 with peer 5f46e19644fd4f4b825c86ba6af768da
I20240503 13:59:53.502812 12776 tablet_bootstrap.cc:492] T cf7323c2594e4cd5967c3d38e4120815 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:53.503108 12776 tablet_bootstrap.cc:492] T cf7323c2594e4cd5967c3d38e4120815 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap complete.
I20240503 13:59:53.503319 12776 ts_tablet_manager.cc:1367] T cf7323c2594e4cd5967c3d38e4120815 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.007s
I20240503 13:59:53.503774 12776 raft_consensus.cc:348] T cf7323c2594e4cd5967c3d38e4120815 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:53.503901 12776 raft_consensus.cc:729] T cf7323c2594e4cd5967c3d38e4120815 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: LEARNER
I20240503 13:59:53.504001 12776 consensus_queue.cc:260] T cf7323c2594e4cd5967c3d38e4120815 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:53.504290 12776 ts_tablet_manager.cc:1398] T cf7323c2594e4cd5967c3d38e4120815 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:53.504766 6942 tablet_copy_service.cc:342] P f58065db70b64ef8b1dc499aa6ae4460: Request end of tablet copy session 8d574fb4a89d4eea947c859470d44dd4-cf7323c2594e4cd5967c3d38e4120815 received from {username='slave'} at 127.0.99.134:57297
I20240503 13:59:53.504853 6942 tablet_copy_service.cc:433] P f58065db70b64ef8b1dc499aa6ae4460: ending tablet copy session 8d574fb4a89d4eea947c859470d44dd4-cf7323c2594e4cd5967c3d38e4120815 on tablet cf7323c2594e4cd5967c3d38e4120815 with peer 8d574fb4a89d4eea947c859470d44dd4
I20240503 13:59:53.573316 9210 consensus_queue.cc:237] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 13:59:53.574558 6790 raft_consensus.cc:1261] T 350ba659df804b95b0692091dc2f68a2 P 4a85e492c8154e9881a9cc2a105d7e64 [term 2 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 13:59:53.574944 12063 consensus_queue.cc:1035] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:53.575707 6657 raft_consensus.cc:1261] T 350ba659df804b95b0692091dc2f68a2 P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 13:59:53.576392 12063 consensus_queue.cc:1035] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:53.577515 12063 raft_consensus.cc:2932] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:53.578773 6657 raft_consensus.cc:2932] T 350ba659df804b95b0692091dc2f68a2 P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:53.580204 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 350ba659df804b95b0692091dc2f68a2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:53.581024 12979 ts_tablet_manager.cc:902] T af8bba02684c4555893def9c1e88ebf0 P 4a85e492c8154e9881a9cc2a105d7e64: Initiating tablet copy from peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:53.581491 12979 tablet_copy_client.cc:308] T af8bba02684c4555893def9c1e88ebf0 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43531
I20240503 13:59:53.582671 6942 tablet_copy_service.cc:140] P f58065db70b64ef8b1dc499aa6ae4460: Received BeginTabletCopySession request for tablet af8bba02684c4555893def9c1e88ebf0 from peer 4a85e492c8154e9881a9cc2a105d7e64 ({username='slave'} at 127.0.99.131:38291)
I20240503 13:59:53.582778 6942 tablet_copy_service.cc:161] P f58065db70b64ef8b1dc499aa6ae4460: Beginning new tablet copy session on tablet af8bba02684c4555893def9c1e88ebf0 from peer 4a85e492c8154e9881a9cc2a105d7e64 at {username='slave'} at 127.0.99.131:38291: session id = 4a85e492c8154e9881a9cc2a105d7e64-af8bba02684c4555893def9c1e88ebf0
I20240503 13:59:53.583456 6790 raft_consensus.cc:2932] T 350ba659df804b95b0692091dc2f68a2 P 4a85e492c8154e9881a9cc2a105d7e64 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:53.585147 6380 catalog_manager.cc:5484] T 350ba659df804b95b0692091dc2f68a2 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index 5 to 7, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:53.585420 6942 tablet_copy_source_session.cc:205] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:53.586298 12979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af8bba02684c4555893def9c1e88ebf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:53.588521 6900 tablet_service.cc:1514] Processing DeleteTablet for tablet 350ba659df804b95b0692091dc2f68a2 with delete_type TABLET_DATA_TOMBSTONED (TS f58065db70b64ef8b1dc499aa6ae4460 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:53.588570 12979 tablet_copy_client.cc:783] T af8bba02684c4555893def9c1e88ebf0 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:53.588685 12979 tablet_copy_client.cc:647] T af8bba02684c4555893def9c1e88ebf0 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:53.588881 12984 tablet_replica.cc:330] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:53.589061 12984 raft_consensus.cc:2229] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:53.589217 12984 raft_consensus.cc:2258] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:53.589893 12979 tablet_copy_client.cc:523] T af8bba02684c4555893def9c1e88ebf0 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:53.590437 12984 ts_tablet_manager.cc:1875] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:53.593015 12979 tablet_bootstrap.cc:492] T af8bba02684c4555893def9c1e88ebf0 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:53.594266 12984 ts_tablet_manager.cc:1888] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 13:59:53.594375 12984 log.cc:1196] T 350ba659df804b95b0692091dc2f68a2 P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/350ba659df804b95b0692091dc2f68a2
I20240503 13:59:53.594920 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet 350ba659df804b95b0692091dc2f68a2 (table rebalance_test_table_1 [id=f757e59886674aafb5f975c30459df7e]) successfully deleted
I20240503 13:59:53.597903 12979 tablet_bootstrap.cc:492] T af8bba02684c4555893def9c1e88ebf0 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:53.598161 12979 tablet_bootstrap.cc:492] T af8bba02684c4555893def9c1e88ebf0 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap complete.
I20240503 13:59:53.598317 12979 ts_tablet_manager.cc:1367] T af8bba02684c4555893def9c1e88ebf0 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 13:59:53.598835 12979 raft_consensus.cc:348] T af8bba02684c4555893def9c1e88ebf0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:53.599102 12979 raft_consensus.cc:729] T af8bba02684c4555893def9c1e88ebf0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: LEARNER
I20240503 13:59:53.599224 12979 consensus_queue.cc:260] T af8bba02684c4555893def9c1e88ebf0 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:53.599545 12979 ts_tablet_manager.cc:1398] T af8bba02684c4555893def9c1e88ebf0 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:53.600180 6942 tablet_copy_service.cc:342] P f58065db70b64ef8b1dc499aa6ae4460: Request end of tablet copy session 4a85e492c8154e9881a9cc2a105d7e64-af8bba02684c4555893def9c1e88ebf0 received from {username='slave'} at 127.0.99.131:38291
I20240503 13:59:53.600292 6942 tablet_copy_service.cc:433] P f58065db70b64ef8b1dc499aa6ae4460: ending tablet copy session 4a85e492c8154e9881a9cc2a105d7e64-af8bba02684c4555893def9c1e88ebf0 on tablet af8bba02684c4555893def9c1e88ebf0 with peer 4a85e492c8154e9881a9cc2a105d7e64
I20240503 13:59:53.614259 12885 ts_tablet_manager.cc:902] T db13f399b9e04fe9a69e729e384d046c P 06ca8dbc004c42b99faa527a4e6f9545: Initiating tablet copy from peer ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:53.614768 12885 tablet_copy_client.cc:308] T db13f399b9e04fe9a69e729e384d046c P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:34091
I20240503 13:59:53.615437 9231 tablet_copy_service.cc:140] P ab1d21935d6049888eb0ceccc3c9d735: Received BeginTabletCopySession request for tablet db13f399b9e04fe9a69e729e384d046c from peer 06ca8dbc004c42b99faa527a4e6f9545 ({username='slave'} at 127.0.99.133:51367)
I20240503 13:59:53.615543 9231 tablet_copy_service.cc:161] P ab1d21935d6049888eb0ceccc3c9d735: Beginning new tablet copy session on tablet db13f399b9e04fe9a69e729e384d046c from peer 06ca8dbc004c42b99faa527a4e6f9545 at {username='slave'} at 127.0.99.133:51367: session id = 06ca8dbc004c42b99faa527a4e6f9545-db13f399b9e04fe9a69e729e384d046c
I20240503 13:59:53.616611 9231 tablet_copy_source_session.cc:205] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:53.617415 12885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db13f399b9e04fe9a69e729e384d046c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:53.619833 12885 tablet_copy_client.cc:783] T db13f399b9e04fe9a69e729e384d046c P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:53.620007 12885 tablet_copy_client.cc:647] T db13f399b9e04fe9a69e729e384d046c P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:53.621990 12885 tablet_copy_client.cc:523] T db13f399b9e04fe9a69e729e384d046c P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:53.622357 12790 ts_tablet_manager.cc:902] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da: Initiating tablet copy from peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:53.622750 12790 tablet_copy_client.cc:308] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:46365
I20240503 13:59:53.623230 12885 tablet_bootstrap.cc:492] T db13f399b9e04fe9a69e729e384d046c P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:53.623232 6811 tablet_copy_service.cc:140] P 4a85e492c8154e9881a9cc2a105d7e64: Received BeginTabletCopySession request for tablet 957b89de328d434d8ea03a59bc93fdcb from peer 5f46e19644fd4f4b825c86ba6af768da ({username='slave'} at 127.0.99.129:57203)
I20240503 13:59:53.623337 6811 tablet_copy_service.cc:161] P 4a85e492c8154e9881a9cc2a105d7e64: Beginning new tablet copy session on tablet 957b89de328d434d8ea03a59bc93fdcb from peer 5f46e19644fd4f4b825c86ba6af768da at {username='slave'} at 127.0.99.129:57203: session id = 5f46e19644fd4f4b825c86ba6af768da-957b89de328d434d8ea03a59bc93fdcb
I20240503 13:59:53.624483 6811 tablet_copy_source_session.cc:205] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:53.625289 12790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 957b89de328d434d8ea03a59bc93fdcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:53.627218 12885 tablet_bootstrap.cc:492] T db13f399b9e04fe9a69e729e384d046c P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:53.627493 12885 tablet_bootstrap.cc:492] T db13f399b9e04fe9a69e729e384d046c P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap complete.
I20240503 13:59:53.627651 12885 ts_tablet_manager.cc:1367] T db13f399b9e04fe9a69e729e384d046c P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20240503 13:59:53.627698 12790 tablet_copy_client.cc:783] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:53.627861 12790 tablet_copy_client.cc:647] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:53.628099 12885 raft_consensus.cc:348] T db13f399b9e04fe9a69e729e384d046c P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 13:59:53.628262 12885 raft_consensus.cc:729] T db13f399b9e04fe9a69e729e384d046c P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: LEARNER
I20240503 13:59:53.628386 12885 consensus_queue.cc:260] T db13f399b9e04fe9a69e729e384d046c P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 13:59:53.628785 12885 ts_tablet_manager.cc:1398] T db13f399b9e04fe9a69e729e384d046c P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:53.631122 12790 tablet_copy_client.cc:523] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:53.631422 9231 tablet_copy_service.cc:342] P ab1d21935d6049888eb0ceccc3c9d735: Request end of tablet copy session 06ca8dbc004c42b99faa527a4e6f9545-db13f399b9e04fe9a69e729e384d046c received from {username='slave'} at 127.0.99.133:51367
I20240503 13:59:53.631495 9231 tablet_copy_service.cc:433] P ab1d21935d6049888eb0ceccc3c9d735: ending tablet copy session 06ca8dbc004c42b99faa527a4e6f9545-db13f399b9e04fe9a69e729e384d046c on tablet db13f399b9e04fe9a69e729e384d046c with peer 06ca8dbc004c42b99faa527a4e6f9545
I20240503 13:59:53.635558 12790 tablet_bootstrap.cc:492] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:53.640475 12790 tablet_bootstrap.cc:492] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:53.640702 12790 tablet_bootstrap.cc:492] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap complete.
I20240503 13:59:53.640882 12790 ts_tablet_manager.cc:1367] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 13:59:53.641268 12790 raft_consensus.cc:348] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:53.641367 12790 raft_consensus.cc:729] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: LEARNER
I20240503 13:59:53.641464 12790 consensus_queue.cc:260] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:53.641824 12790 ts_tablet_manager.cc:1398] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:53.642588 6811 tablet_copy_service.cc:342] P 4a85e492c8154e9881a9cc2a105d7e64: Request end of tablet copy session 5f46e19644fd4f4b825c86ba6af768da-957b89de328d434d8ea03a59bc93fdcb received from {username='slave'} at 127.0.99.129:57203
I20240503 13:59:53.642652 6811 tablet_copy_service.cc:433] P 4a85e492c8154e9881a9cc2a105d7e64: ending tablet copy session 5f46e19644fd4f4b825c86ba6af768da-957b89de328d434d8ea03a59bc93fdcb on tablet 957b89de328d434d8ea03a59bc93fdcb with peer 5f46e19644fd4f4b825c86ba6af768da
I20240503 13:59:53.657027 6528 raft_consensus.cc:1203] T 3286ea636a5a4d23bf92e8452afd608d P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:53.660171 9081 raft_consensus.cc:1203] T f9d875f4872e4f5da12a326381d523c0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:53.661303 12790 ts_tablet_manager.cc:902] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 5f46e19644fd4f4b825c86ba6af768da: Initiating tablet copy from peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:53.661705 12790 tablet_copy_client.cc:308] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:46365
I20240503 13:59:53.662308 6811 tablet_copy_service.cc:140] P 4a85e492c8154e9881a9cc2a105d7e64: Received BeginTabletCopySession request for tablet c8b2d8a9ed5f4b09812a85445a8e3b85 from peer 5f46e19644fd4f4b825c86ba6af768da ({username='slave'} at 127.0.99.129:57203)
I20240503 13:59:53.662422 6811 tablet_copy_service.cc:161] P 4a85e492c8154e9881a9cc2a105d7e64: Beginning new tablet copy session on tablet c8b2d8a9ed5f4b09812a85445a8e3b85 from peer 5f46e19644fd4f4b825c86ba6af768da at {username='slave'} at 127.0.99.129:57203: session id = 5f46e19644fd4f4b825c86ba6af768da-c8b2d8a9ed5f4b09812a85445a8e3b85
I20240503 13:59:53.663326 6811 tablet_copy_source_session.cc:205] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:53.664713 12790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8b2d8a9ed5f4b09812a85445a8e3b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:53.666548 12790 tablet_copy_client.cc:783] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:53.666713 12790 tablet_copy_client.cc:647] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:53.669400 12790 tablet_copy_client.cc:523] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:53.670814 12790 tablet_bootstrap.cc:492] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:53.674716 12790 tablet_bootstrap.cc:492] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:53.674949 12790 tablet_bootstrap.cc:492] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap complete.
I20240503 13:59:53.675139 12790 ts_tablet_manager.cc:1367] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 13:59:53.675627 12790 raft_consensus.cc:348] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:53.675746 12790 raft_consensus.cc:729] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: LEARNER
I20240503 13:59:53.675837 12790 consensus_queue.cc:260] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:53.676090 12790 ts_tablet_manager.cc:1398] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:53.676950 8948 raft_consensus.cc:1203] T f3dd70554b644cddb1e8eefe7e4ea646 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:53.678540 6811 tablet_copy_service.cc:342] P 4a85e492c8154e9881a9cc2a105d7e64: Request end of tablet copy session 5f46e19644fd4f4b825c86ba6af768da-c8b2d8a9ed5f4b09812a85445a8e3b85 received from {username='slave'} at 127.0.99.129:57203
I20240503 13:59:53.678617 6811 tablet_copy_service.cc:433] P 4a85e492c8154e9881a9cc2a105d7e64: ending tablet copy session 5f46e19644fd4f4b825c86ba6af768da-c8b2d8a9ed5f4b09812a85445a8e3b85 on tablet c8b2d8a9ed5f4b09812a85445a8e3b85 with peer 5f46e19644fd4f4b825c86ba6af768da
I20240503 13:59:53.679579 6527 raft_consensus.cc:1203] T 8482be0fb2784324b77b272e40f57d6b P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:53.710093 8948 consensus_queue.cc:237] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:53.711555 9081 raft_consensus.cc:1261] T a16acc33ab6c45858c52cceb4355207a P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:53.711671 9343 raft_consensus.cc:1261] T a16acc33ab6c45858c52cceb4355207a P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:53.711839 8884 consensus_peers.cc:485] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): Couldn't send request to peer 2145d15ab2d74c0287839266f1411a43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a16acc33ab6c45858c52cceb4355207a. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:53.712005 11750 consensus_queue.cc:1035] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:53.712493 11750 consensus_queue.cc:1035] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:53.713707 13015 raft_consensus.cc:2932] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:53.715111 9343 raft_consensus.cc:2932] T a16acc33ab6c45858c52cceb4355207a P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:53.715065 9081 raft_consensus.cc:2932] T a16acc33ab6c45858c52cceb4355207a P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:53.718376 6381 catalog_manager.cc:5484] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:53.748067 9079 raft_consensus.cc:1203] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:53.763031 6660 raft_consensus.cc:1203] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:53.769788 9210 raft_consensus.cc:1203] T 98466061b550450bb9d0041427f6f19d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:53.796301 6921 raft_consensus.cc:1203] T 64135eb6ef574c95b3677e32d66748a4 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:53.849619 9079 raft_consensus.cc:1203] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:53.888020 9343 raft_consensus.cc:1203] T 1ab63490264c4755aeab897f865e9a11 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:53.892390 6921 raft_consensus.cc:1203] T e2d398cb36d54bc785648751b5b52370 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:53.902592 6527 raft_consensus.cc:1203] T e92eaea6e9c8493690b749f162949ac4 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:53.934316 9079 raft_consensus.cc:1203] T cf7323c2594e4cd5967c3d38e4120815 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:53.978757 11471 raft_consensus.cc:1050] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735: attempting to promote NON_VOTER 2145d15ab2d74c0287839266f1411a43 to VOTER
I20240503 13:59:53.979173 11471 consensus_queue.cc:237] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 13:59:53.980496 6660 raft_consensus.cc:1261] T f4cba4ca296e407c9520a6eb7419ed9d P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:53.980643 8948 raft_consensus.cc:1261] T f4cba4ca296e407c9520a6eb7419ed9d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:53.981550 6527 raft_consensus.cc:1261] T f4cba4ca296e407c9520a6eb7419ed9d P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:53.982548 12243 consensus_queue.cc:1035] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:53.983064 11665 consensus_queue.cc:1035] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:53.983259 12243 consensus_queue.cc:1035] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:53.985632 11471 raft_consensus.cc:2932] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:53.986222 8948 raft_consensus.cc:2932] T f4cba4ca296e407c9520a6eb7419ed9d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:53.987516 6527 raft_consensus.cc:2932] T f4cba4ca296e407c9520a6eb7419ed9d P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:53.987793 6657 raft_consensus.cc:2932] T f4cba4ca296e407c9520a6eb7419ed9d P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:53.990612 6381 catalog_manager.cc:5484] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:53.993888 9205 consensus_queue.cc:237] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 13:59:53.994895 6657 raft_consensus.cc:1261] T f4cba4ca296e407c9520a6eb7419ed9d P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:53.995250 12063 consensus_queue.cc:1035] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:53.995316 6527 raft_consensus.cc:1261] T f4cba4ca296e407c9520a6eb7419ed9d P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:53.995935 12063 consensus_queue.cc:1035] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:53.997502 6790 raft_consensus.cc:1203] T af8bba02684c4555893def9c1e88ebf0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:53.997422 12243 raft_consensus.cc:2932] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:53.998097 6527 raft_consensus.cc:2932] T f4cba4ca296e407c9520a6eb7419ed9d P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:53.998805 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f4cba4ca296e407c9520a6eb7419ed9d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:53.998931 6657 raft_consensus.cc:2932] T f4cba4ca296e407c9520a6eb7419ed9d P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:54.004279 6380 catalog_manager.cc:5484] T f4cba4ca296e407c9520a6eb7419ed9d P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:54.007069 8926 tablet_service.cc:1514] Processing DeleteTablet for tablet f4cba4ca296e407c9520a6eb7419ed9d with delete_type TABLET_DATA_TOMBSTONED (TS 06ca8dbc004c42b99faa527a4e6f9545 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:54.007472 13164 tablet_replica.cc:330] T f4cba4ca296e407c9520a6eb7419ed9d P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:54.007635 13164 raft_consensus.cc:2229] T f4cba4ca296e407c9520a6eb7419ed9d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:54.007792 13164 raft_consensus.cc:2258] T f4cba4ca296e407c9520a6eb7419ed9d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:54.015854 13164 ts_tablet_manager.cc:1875] T f4cba4ca296e407c9520a6eb7419ed9d P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:54.018095 13164 ts_tablet_manager.cc:1888] T f4cba4ca296e407c9520a6eb7419ed9d P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:54.018187 13164 log.cc:1196] T f4cba4ca296e407c9520a6eb7419ed9d P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/f4cba4ca296e407c9520a6eb7419ed9d
I20240503 13:59:54.018795 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet f4cba4ca296e407c9520a6eb7419ed9d (table rebalancer_extra_table_34 [id=22ad55d53adb4db5b7dd551f6384d11b]) successfully deleted
I20240503 13:59:54.086275 6527 raft_consensus.cc:1203] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:54.125959 6527 raft_consensus.cc:1203] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:54.145972 12156 raft_consensus.cc:1050] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da: attempting to promote NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 to VOTER
I20240503 13:59:54.146538 12156 consensus_queue.cc:237] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 13:59:54.148039 6921 raft_consensus.cc:1261] T f9d875f4872e4f5da12a326381d523c0 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:54.148226 9079 raft_consensus.cc:1261] T f9d875f4872e4f5da12a326381d523c0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:54.148239 6790 raft_consensus.cc:1261] T f9d875f4872e4f5da12a326381d523c0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:54.148643 12508 consensus_queue.cc:1035] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:54.148808 13157 consensus_queue.cc:1035] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:54.148934 13226 consensus_queue.cc:1035] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:54.152601 12508 raft_consensus.cc:2932] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.154767 6921 raft_consensus.cc:2932] T f9d875f4872e4f5da12a326381d523c0 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.154753 9079 raft_consensus.cc:2932] T f9d875f4872e4f5da12a326381d523c0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.156651 6790 raft_consensus.cc:2932] T f9d875f4872e4f5da12a326381d523c0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.158432 6380 catalog_manager.cc:5484] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index 4 to 5, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:54.163910 6527 consensus_queue.cc:237] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 13:59:54.165339 6790 raft_consensus.cc:1261] T f9d875f4872e4f5da12a326381d523c0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:54.165699 12156 consensus_queue.cc:1035] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:54.167491 13157 raft_consensus.cc:2932] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.169528 9079 raft_consensus.cc:1261] T f9d875f4872e4f5da12a326381d523c0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:54.169904 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f9d875f4872e4f5da12a326381d523c0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:54.169958 12156 consensus_queue.cc:1035] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:54.170562 6790 raft_consensus.cc:2932] T f9d875f4872e4f5da12a326381d523c0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.170984 6387 catalog_manager.cc:5484] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index 5 to 6, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:54.173419 9081 raft_consensus.cc:2932] T f9d875f4872e4f5da12a326381d523c0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.173942 6900 tablet_service.cc:1514] Processing DeleteTablet for tablet f9d875f4872e4f5da12a326381d523c0 with delete_type TABLET_DATA_TOMBSTONED (TS f58065db70b64ef8b1dc499aa6ae4460 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:54.176837 13235 tablet_replica.cc:330] T f9d875f4872e4f5da12a326381d523c0 P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:54.177064 13235 raft_consensus.cc:2229] T f9d875f4872e4f5da12a326381d523c0 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:54.177263 13235 raft_consensus.cc:2258] T f9d875f4872e4f5da12a326381d523c0 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:54.177978 13235 ts_tablet_manager.cc:1875] T f9d875f4872e4f5da12a326381d523c0 P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:54.180150 11378 raft_consensus.cc:1050] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6: attempting to promote NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 to VOTER
I20240503 13:59:54.180495 13235 ts_tablet_manager.cc:1888] T f9d875f4872e4f5da12a326381d523c0 P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:54.180575 13235 log.cc:1196] T f9d875f4872e4f5da12a326381d523c0 P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/f9d875f4872e4f5da12a326381d523c0
I20240503 13:59:54.180529 11378 consensus_queue.cc:237] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 13:59:54.181381 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet f9d875f4872e4f5da12a326381d523c0 (table rebalance_test_table_19 [id=aaf095359a4a40ecbae916f4245a8f5f]) successfully deleted
I20240503 13:59:54.181882 9081 raft_consensus.cc:1261] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.182161 6921 raft_consensus.cc:1261] T 47d38e1d60dd42eabf7191598ded8cd5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.182216 13080 consensus_queue.cc:1035] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.182698 13080 consensus_queue.cc:1035] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.182744 9205 raft_consensus.cc:1261] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.183429 13080 consensus_queue.cc:1035] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.186017 6921 raft_consensus.cc:2932] T 47d38e1d60dd42eabf7191598ded8cd5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.186120 9205 raft_consensus.cc:2932] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.185688 11378 raft_consensus.cc:2932] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.189688 6387 catalog_manager.cc:5484] T 47d38e1d60dd42eabf7191598ded8cd5 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.191483 8946 raft_consensus.cc:1203] T db13f399b9e04fe9a69e729e384d046c P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:54.192222 9081 raft_consensus.cc:2932] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.197803 9342 consensus_queue.cc:237] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 13:59:54.199807 9081 raft_consensus.cc:1261] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.199828 9205 raft_consensus.cc:1261] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.200542 11167 consensus_queue.cc:1035] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.200748 13080 consensus_queue.cc:1035] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.202431 11167 raft_consensus.cc:2932] T 47d38e1d60dd42eabf7191598ded8cd5 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.202827 9205 raft_consensus.cc:2932] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.203181 9081 raft_consensus.cc:2932] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.204928 6374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 47d38e1d60dd42eabf7191598ded8cd5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:54.206504 6381 catalog_manager.cc:5484] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index 3 to 4, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.209461 6900 tablet_service.cc:1514] Processing DeleteTablet for tablet 47d38e1d60dd42eabf7191598ded8cd5 with delete_type TABLET_DATA_TOMBSTONED (TS f58065db70b64ef8b1dc499aa6ae4460 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:54.209692 13235 tablet_replica.cc:330] T 47d38e1d60dd42eabf7191598ded8cd5 P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:54.209856 13235 raft_consensus.cc:2229] T 47d38e1d60dd42eabf7191598ded8cd5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:54.209975 13235 raft_consensus.cc:2258] T 47d38e1d60dd42eabf7191598ded8cd5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:54.210523 13235 ts_tablet_manager.cc:1875] T 47d38e1d60dd42eabf7191598ded8cd5 P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:54.212898 13235 ts_tablet_manager.cc:1888] T 47d38e1d60dd42eabf7191598ded8cd5 P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:54.212965 13235 log.cc:1196] T 47d38e1d60dd42eabf7191598ded8cd5 P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/47d38e1d60dd42eabf7191598ded8cd5
I20240503 13:59:54.213555 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet 47d38e1d60dd42eabf7191598ded8cd5 (table rebalancer_extra_table_58 [id=2f45bc2e41c6493a85eab02853317fd7]) successfully deleted
I20240503 13:59:54.223242 13157 raft_consensus.cc:1050] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da: attempting to promote NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 to VOTER
I20240503 13:59:54.223735 13157 consensus_queue.cc:237] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
I20240503 13:59:54.224822 11750 raft_consensus.cc:1050] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545: attempting to promote NON_VOTER 5f46e19644fd4f4b825c86ba6af768da to VOTER
I20240503 13:59:54.225229 11750 consensus_queue.cc:237] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 13:59:54.225293 9342 raft_consensus.cc:1261] T f3dd70554b644cddb1e8eefe7e4ea646 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.225313 6657 raft_consensus.cc:1261] T f3dd70554b644cddb1e8eefe7e4ea646 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.226094 12508 consensus_queue.cc:1035] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.226876 6660 raft_consensus.cc:1261] T 3286ea636a5a4d23bf92e8452afd608d P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.227087 6528 raft_consensus.cc:1261] T 3286ea636a5a4d23bf92e8452afd608d P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.227087 8946 raft_consensus.cc:1261] T f3dd70554b644cddb1e8eefe7e4ea646 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.227479 12367 consensus_queue.cc:1035] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.227593 9205 raft_consensus.cc:1261] T 3286ea636a5a4d23bf92e8452afd608d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.228039 12367 consensus_queue.cc:1035] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.228166 12156 consensus_queue.cc:1035] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.228577 12367 consensus_queue.cc:1035] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.228627 12508 consensus_queue.cc:1035] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.231880 12367 raft_consensus.cc:2932] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.232611 9205 raft_consensus.cc:2932] T 3286ea636a5a4d23bf92e8452afd608d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.234043 6658 raft_consensus.cc:2932] T 3286ea636a5a4d23bf92e8452afd608d P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.234411 6528 raft_consensus.cc:2932] T 3286ea636a5a4d23bf92e8452afd608d P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.236024 6387 catalog_manager.cc:5484] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:54.236491 12156 raft_consensus.cc:2932] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:54.238526 6657 raft_consensus.cc:2932] T f3dd70554b644cddb1e8eefe7e4ea646 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:54.239605 12528 raft_consensus.cc:1050] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64: attempting to promote NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 to VOTER
I20240503 13:59:54.240096 12528 consensus_queue.cc:237] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 13:59:54.240031 8946 raft_consensus.cc:2932] T f3dd70554b644cddb1e8eefe7e4ea646 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:54.241463 6380 catalog_manager.cc:5484] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index 2 to 3, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:54.241995 9081 raft_consensus.cc:1261] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.243853 6921 raft_consensus.cc:1261] T 64135eb6ef574c95b3677e32d66748a4 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.244341 13116 consensus_queue.cc:1035] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.244679 13116 consensus_queue.cc:1035] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.245062 9342 raft_consensus.cc:1261] T 64135eb6ef574c95b3677e32d66748a4 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.245909 12032 consensus_queue.cc:1035] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.247467 13158 raft_consensus.cc:1050] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4: attempting to promote NON_VOTER 5f46e19644fd4f4b825c86ba6af768da to VOTER
I20240503 13:59:54.247443 6526 consensus_queue.cc:237] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
I20240503 13:59:54.247898 13158 consensus_queue.cc:237] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 13:59:54.249644 8946 raft_consensus.cc:1261] T 8482be0fb2784324b77b272e40f57d6b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.250141 12864 consensus_queue.cc:1035] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.250298 9341 raft_consensus.cc:1261] T 8482be0fb2784324b77b272e40f57d6b P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.250303 6658 raft_consensus.cc:1261] T f3dd70554b644cddb1e8eefe7e4ea646 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.250661 12864 consensus_queue.cc:1035] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.251997 13157 consensus_queue.cc:1035] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.254961 8948 raft_consensus.cc:1261] T f3dd70554b644cddb1e8eefe7e4ea646 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.255429 13157 consensus_queue.cc:1035] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.256512 13157 raft_consensus.cc:1050] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da: attempting to promote NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 to VOTER
I20240503 13:59:54.256870 13157 consensus_queue.cc:237] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 13:59:54.256984 8947 consensus_queue.cc:237] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 13:59:54.257372 12508 raft_consensus.cc:2932] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:54.258183 8947 raft_consensus.cc:2932] T f3dd70554b644cddb1e8eefe7e4ea646 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:54.258900 6657 raft_consensus.cc:1261] T 3286ea636a5a4d23bf92e8452afd608d P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.259460 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f3dd70554b644cddb1e8eefe7e4ea646 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:54.259708 11750 consensus_queue.cc:1035] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.260013 6527 raft_consensus.cc:1261] T 3286ea636a5a4d23bf92e8452afd608d P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.260282 9079 raft_consensus.cc:1261] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:54.260280 6657 raft_consensus.cc:1261] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:54.260288 6921 raft_consensus.cc:1261] T 39abc23bb3024e699e2f5bdb3bf5a7ea P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:54.260646 12156 consensus_queue.cc:1035] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:54.261139 12156 consensus_queue.cc:1035] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:54.261276 13226 consensus_queue.cc:1035] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:54.261401 11750 consensus_queue.cc:1035] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.261724 6526 raft_consensus.cc:1261] T 8482be0fb2784324b77b272e40f57d6b P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.262241 13280 consensus_queue.cc:1035] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.264343 6381 catalog_manager.cc:5484] T f3dd70554b644cddb1e8eefe7e4ea646 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:54.265594 6658 raft_consensus.cc:2932] T f3dd70554b644cddb1e8eefe7e4ea646 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:54.266218 11750 raft_consensus.cc:2932] T 3286ea636a5a4d23bf92e8452afd608d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.266623 6657 raft_consensus.cc:2932] T 3286ea636a5a4d23bf92e8452afd608d P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.267242 13157 raft_consensus.cc:2932] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.269219 6527 raft_consensus.cc:2932] T 3286ea636a5a4d23bf92e8452afd608d P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.270654 9321 tablet_service.cc:1514] Processing DeleteTablet for tablet f3dd70554b644cddb1e8eefe7e4ea646 with delete_type TABLET_DATA_TOMBSTONED (TS 0d77752181e944698e809e325e5937e6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:54.270929 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3286ea636a5a4d23bf92e8452afd608d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:54.271198 6657 raft_consensus.cc:2932] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.272217 9081 raft_consensus.cc:2932] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.273792 6380 catalog_manager.cc:5484] T 3286ea636a5a4d23bf92e8452afd608d P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.274199 12032 raft_consensus.cc:2932] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:54.276876 6920 raft_consensus.cc:2932] T 39abc23bb3024e699e2f5bdb3bf5a7ea P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.277112 13280 raft_consensus.cc:2932] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.278534 9081 raft_consensus.cc:2932] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:54.278951 6386 catalog_manager.cc:5484] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index 4 to 5, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.279402 9192 tablet_service.cc:1514] Processing DeleteTablet for tablet 3286ea636a5a4d23bf92e8452afd608d with delete_type TABLET_DATA_TOMBSTONED (TS ab1d21935d6049888eb0ceccc3c9d735 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57960
I20240503 13:59:54.279654 13015 raft_consensus.cc:1050] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545: attempting to promote NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 to VOTER
I20240503 13:59:54.280051 13015 consensus_queue.cc:237] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 13:59:54.280184 8946 raft_consensus.cc:2932] T 8482be0fb2784324b77b272e40f57d6b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.281147 13294 tablet_replica.cc:330] T f3dd70554b644cddb1e8eefe7e4ea646 P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:54.281296 13294 raft_consensus.cc:2229] T f3dd70554b644cddb1e8eefe7e4ea646 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:54.281459 6790 raft_consensus.cc:1261] T e2d398cb36d54bc785648751b5b52370 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.281955 12367 consensus_queue.cc:1035] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.282641 6526 raft_consensus.cc:1261] T e2d398cb36d54bc785648751b5b52370 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.281409 13294 pending_rounds.cc:62] T f3dd70554b644cddb1e8eefe7e4ea646 P 0d77752181e944698e809e325e5937e6: Trying to abort 1 pending ops.
I20240503 13:59:54.283046 13294 pending_rounds.cc:68] T f3dd70554b644cddb1e8eefe7e4ea646 P 0d77752181e944698e809e325e5937e6: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023594677139427328 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "f3dd70554b644cddb1e8eefe7e4ea646" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } } }
I20240503 13:59:54.283221 13294 raft_consensus.cc:2894] T f3dd70554b644cddb1e8eefe7e4ea646 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240503 13:59:54.283346 13294 raft_consensus.cc:2866] T f3dd70554b644cddb1e8eefe7e4ea646 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 13:59:54.283403 13294 raft_consensus.cc:2258] T f3dd70554b644cddb1e8eefe7e4ea646 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:54.283376 6527 raft_consensus.cc:2932] T 8482be0fb2784324b77b272e40f57d6b P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.283473 9342 raft_consensus.cc:2932] T 8482be0fb2784324b77b272e40f57d6b P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.284106 11750 consensus_queue.cc:1035] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.287695 12367 raft_consensus.cc:1050] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545: attempting to promote NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 to VOTER
I20240503 13:59:54.288055 12367 consensus_queue.cc:237] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 13:59:54.288684 6922 raft_consensus.cc:1261] T e2d398cb36d54bc785648751b5b52370 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.288586 6381 catalog_manager.cc:5484] T 8482be0fb2784324b77b272e40f57d6b P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.289700 9342 raft_consensus.cc:2932] T 64135eb6ef574c95b3677e32d66748a4 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:54.290939 9081 raft_consensus.cc:1261] T 98466061b550450bb9d0041427f6f19d P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.291657 11750 consensus_queue.cc:1035] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.291975 11750 consensus_queue.cc:1035] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.292539 9205 raft_consensus.cc:1261] T 98466061b550450bb9d0041427f6f19d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.293130 12367 consensus_queue.cc:1035] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.293615 6920 raft_consensus.cc:2932] T 64135eb6ef574c95b3677e32d66748a4 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:54.293992 6386 catalog_manager.cc:5484] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:54.295847 9342 raft_consensus.cc:1261] T 98466061b550450bb9d0041427f6f19d P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.296205 13015 consensus_queue.cc:1035] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.300488 9079 consensus_queue.cc:237] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 13:59:54.299216 6527 consensus_queue.cc:237] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 13:59:54.301796 9079 raft_consensus.cc:1261] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:54.301975 6920 raft_consensus.cc:1261] T 39abc23bb3024e699e2f5bdb3bf5a7ea P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:54.301968 6527 raft_consensus.cc:1261] T 8482be0fb2784324b77b272e40f57d6b P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.302332 13289 consensus_queue.cc:1035] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:54.302517 12864 consensus_queue.cc:1035] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.302645 9342 raft_consensus.cc:1261] T 8482be0fb2784324b77b272e40f57d6b P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.302647 13289 consensus_queue.cc:1035] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:54.303231 12864 consensus_queue.cc:1035] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.305405 12864 raft_consensus.cc:2932] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.305948 6527 raft_consensus.cc:2932] T 8482be0fb2784324b77b272e40f57d6b P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.307088 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8482be0fb2784324b77b272e40f57d6b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:54.307098 9342 raft_consensus.cc:2932] T 8482be0fb2784324b77b272e40f57d6b P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.308560 6386 catalog_manager.cc:5484] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:54.309181 12508 raft_consensus.cc:2932] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.311734 8926 tablet_service.cc:1514] Processing DeleteTablet for tablet 8482be0fb2784324b77b272e40f57d6b with delete_type TABLET_DATA_TOMBSTONED (TS 06ca8dbc004c42b99faa527a4e6f9545 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:54.311873 13295 tablet_replica.cc:330] T 3286ea636a5a4d23bf92e8452afd608d P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 13:59:54.311931 13164 tablet_replica.cc:330] T 8482be0fb2784324b77b272e40f57d6b P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:54.312013 13295 raft_consensus.cc:2229] T 3286ea636a5a4d23bf92e8452afd608d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:54.312047 13164 raft_consensus.cc:2229] T 8482be0fb2784324b77b272e40f57d6b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:54.312144 13295 raft_consensus.cc:2258] T 3286ea636a5a4d23bf92e8452afd608d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:54.312152 13164 raft_consensus.cc:2258] T 8482be0fb2784324b77b272e40f57d6b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:54.312781 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 39abc23bb3024e699e2f5bdb3bf5a7ea with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:54.312827 13295 ts_tablet_manager.cc:1875] T 3286ea636a5a4d23bf92e8452afd608d P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:54.313097 13015 raft_consensus.cc:2932] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:54.313277 9079 raft_consensus.cc:2932] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
W20240503 13:59:54.313382 13272 consensus_queue.cc:1159] T f3dd70554b644cddb1e8eefe7e4ea646 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0d77752181e944698e809e325e5937e6" status { last_received { term: 1 index: 3 } last_committed_idx: 2 last_received_current_leader { term: 1 index: 3 } }
I20240503 13:59:54.314141 6922 raft_consensus.cc:2932] T 39abc23bb3024e699e2f5bdb3bf5a7ea P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.316053 13294 ts_tablet_manager.cc:1875] T f3dd70554b644cddb1e8eefe7e4ea646 P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:54.316319 11750 raft_consensus.cc:2932] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:54.318087 9205 raft_consensus.cc:2932] T 98466061b550450bb9d0041427f6f19d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:54.318238 6790 raft_consensus.cc:2932] T e2d398cb36d54bc785648751b5b52370 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:54.320371 13294 ts_tablet_manager.cc:1888] T f3dd70554b644cddb1e8eefe7e4ea646 P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:54.320477 13294 log.cc:1196] T f3dd70554b644cddb1e8eefe7e4ea646 P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/f3dd70554b644cddb1e8eefe7e4ea646
I20240503 13:59:54.321493 6381 catalog_manager.cc:5484] T e2d398cb36d54bc785648751b5b52370 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:54.322918 13295 ts_tablet_manager.cc:1888] T 3286ea636a5a4d23bf92e8452afd608d P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:54.323042 13295 log.cc:1196] T 3286ea636a5a4d23bf92e8452afd608d P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/3286ea636a5a4d23bf92e8452afd608d
I20240503 13:59:54.323740 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet f3dd70554b644cddb1e8eefe7e4ea646 (table rebalancer_extra_table_58 [id=2f45bc2e41c6493a85eab02853317fd7]) successfully deleted
I20240503 13:59:54.323645 6922 raft_consensus.cc:2932] T e2d398cb36d54bc785648751b5b52370 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:54.324558 6386 catalog_manager.cc:5484] T 98466061b550450bb9d0041427f6f19d P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index 2 to 3, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:54.324752 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet 3286ea636a5a4d23bf92e8452afd608d (table rebalancer_extra_table_6 [id=48b6af18e5404209abb657e70e9adedb]) successfully deleted
I20240503 13:59:54.325359 9343 raft_consensus.cc:2932] T 98466061b550450bb9d0041427f6f19d P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:54.326568 6387 catalog_manager.cc:5484] T 39abc23bb3024e699e2f5bdb3bf5a7ea P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.328614 6526 raft_consensus.cc:2932] T e2d398cb36d54bc785648751b5b52370 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:54.330783 13164 ts_tablet_manager.cc:1875] T 8482be0fb2784324b77b272e40f57d6b P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:54.331260 9081 raft_consensus.cc:2932] T 98466061b550450bb9d0041427f6f19d P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:54.334151 13164 ts_tablet_manager.cc:1888] T 8482be0fb2784324b77b272e40f57d6b P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:54.334223 13164 log.cc:1196] T 8482be0fb2784324b77b272e40f57d6b P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/8482be0fb2784324b77b272e40f57d6b
I20240503 13:59:54.334928 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 8482be0fb2784324b77b272e40f57d6b (table rebalancer_extra_table_36 [id=25d54e91bed8402eb84d2c36a0601777]) successfully deleted
I20240503 13:59:54.335441 13306 ts_tablet_manager.cc:902] T a16acc33ab6c45858c52cceb4355207a P 2145d15ab2d74c0287839266f1411a43: Initiating tablet copy from peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:54.335968 13306 tablet_copy_client.cc:308] T a16acc33ab6c45858c52cceb4355207a P 2145d15ab2d74c0287839266f1411a43: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:34547
I20240503 13:59:54.336411 8968 tablet_copy_service.cc:140] P 06ca8dbc004c42b99faa527a4e6f9545: Received BeginTabletCopySession request for tablet a16acc33ab6c45858c52cceb4355207a from peer 2145d15ab2d74c0287839266f1411a43 ({username='slave'} at 127.0.99.130:39847)
I20240503 13:59:54.336514 8968 tablet_copy_service.cc:161] P 06ca8dbc004c42b99faa527a4e6f9545: Beginning new tablet copy session on tablet a16acc33ab6c45858c52cceb4355207a from peer 2145d15ab2d74c0287839266f1411a43 at {username='slave'} at 127.0.99.130:39847: session id = 2145d15ab2d74c0287839266f1411a43-a16acc33ab6c45858c52cceb4355207a
I20240503 13:59:54.337409 8968 tablet_copy_source_session.cc:205] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:54.338064 6639 tablet_service.cc:1514] Processing DeleteTablet for tablet 39abc23bb3024e699e2f5bdb3bf5a7ea with delete_type TABLET_DATA_TOMBSTONED (TS 2145d15ab2d74c0287839266f1411a43 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:54.339203 13306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a16acc33ab6c45858c52cceb4355207a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:54.342978 8946 consensus_queue.cc:237] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 13:59:54.344362 11167 raft_consensus.cc:1050] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6: attempting to promote NON_VOTER 5f46e19644fd4f4b825c86ba6af768da to VOTER
I20240503 13:59:54.344415 9205 raft_consensus.cc:1261] T 98466061b550450bb9d0041427f6f19d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.344770 11167 consensus_queue.cc:237] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 13:59:54.344925 13015 consensus_queue.cc:1035] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.345364 13306 tablet_copy_client.cc:783] T a16acc33ab6c45858c52cceb4355207a P 2145d15ab2d74c0287839266f1411a43: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:54.345569 13306 tablet_copy_client.cc:647] T a16acc33ab6c45858c52cceb4355207a P 2145d15ab2d74c0287839266f1411a43: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:54.346055 9081 raft_consensus.cc:1261] T 98466061b550450bb9d0041427f6f19d P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.346416 9210 raft_consensus.cc:1261] T e92eaea6e9c8493690b749f162949ac4 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.346585 8946 raft_consensus.cc:1261] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.346683 13015 consensus_queue.cc:1035] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.347064 11378 consensus_queue.cc:1035] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.347019 13015 raft_consensus.cc:2932] T 98466061b550450bb9d0041427f6f19d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:54.347375 13242 consensus_queue.cc:1035] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.348699 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 98466061b550450bb9d0041427f6f19d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:54.348893 6526 raft_consensus.cc:1261] T e92eaea6e9c8493690b749f162949ac4 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.349095 9205 raft_consensus.cc:2932] T 98466061b550450bb9d0041427f6f19d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:54.349273 13242 consensus_queue.cc:1035] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.349869 8947 consensus_queue.cc:237] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 13:59:54.352381 6385 catalog_manager.cc:5484] T 98466061b550450bb9d0041427f6f19d P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:54.354177 6922 raft_consensus.cc:1261] T e2d398cb36d54bc785648751b5b52370 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.354239 6526 raft_consensus.cc:1261] T e2d398cb36d54bc785648751b5b52370 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.354624 11750 consensus_queue.cc:1035] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.354882 9321 tablet_service.cc:1514] Processing DeleteTablet for tablet 98466061b550450bb9d0041427f6f19d with delete_type TABLET_DATA_TOMBSTONED (TS 0d77752181e944698e809e325e5937e6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:54.354969 11750 consensus_queue.cc:1035] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.355418 13294 tablet_replica.cc:330] T 98466061b550450bb9d0041427f6f19d P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:54.355566 13294 raft_consensus.cc:2229] T 98466061b550450bb9d0041427f6f19d P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:54.355691 13294 raft_consensus.cc:2258] T 98466061b550450bb9d0041427f6f19d P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:54.355979 13306 tablet_copy_client.cc:523] T a16acc33ab6c45858c52cceb4355207a P 2145d15ab2d74c0287839266f1411a43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:54.357275 11750 raft_consensus.cc:2932] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:54.357743 9081 raft_consensus.cc:2932] T 98466061b550450bb9d0041427f6f19d P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:54.359004 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e2d398cb36d54bc785648751b5b52370 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:54.359690 6922 raft_consensus.cc:2932] T e2d398cb36d54bc785648751b5b52370 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:54.361856 6526 raft_consensus.cc:2932] T e2d398cb36d54bc785648751b5b52370 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:54.362512 6385 catalog_manager.cc:5484] T e2d398cb36d54bc785648751b5b52370 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index 3 to 4, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:54.362684 13306 tablet_bootstrap.cc:492] T a16acc33ab6c45858c52cceb4355207a P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:54.366092 13315 tablet_replica.cc:330] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:54.366143 13294 ts_tablet_manager.cc:1875] T 98466061b550450bb9d0041427f6f19d P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:54.366317 6769 tablet_service.cc:1514] Processing DeleteTablet for tablet e2d398cb36d54bc785648751b5b52370 with delete_type TABLET_DATA_TOMBSTONED (TS 4a85e492c8154e9881a9cc2a105d7e64 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:54.366326 13315 raft_consensus.cc:2229] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:54.366571 13315 raft_consensus.cc:2258] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:54.368247 13315 ts_tablet_manager.cc:1875] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:54.369470 13294 ts_tablet_manager.cc:1888] T 98466061b550450bb9d0041427f6f19d P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:54.369545 13294 log.cc:1196] T 98466061b550450bb9d0041427f6f19d P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/98466061b550450bb9d0041427f6f19d
I20240503 13:59:54.370066 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet 98466061b550450bb9d0041427f6f19d (table rebalancer_extra_table_50 [id=483a5fbd5c7a44c8bd3bb20336c11e18]) successfully deleted
I20240503 13:59:54.371102 13315 ts_tablet_manager.cc:1888] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:54.371165 13315 log.cc:1196] T 39abc23bb3024e699e2f5bdb3bf5a7ea P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/39abc23bb3024e699e2f5bdb3bf5a7ea
I20240503 13:59:54.371930 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet 39abc23bb3024e699e2f5bdb3bf5a7ea (table rebalance_test_table_17 [id=f8de54141608449ab9dffc176985cb69]) successfully deleted
I20240503 13:59:54.372980 13323 tablet_replica.cc:330] T e2d398cb36d54bc785648751b5b52370 P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:54.373076 13306 tablet_bootstrap.cc:492] T a16acc33ab6c45858c52cceb4355207a P 2145d15ab2d74c0287839266f1411a43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:54.372928 13242 raft_consensus.cc:2932] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.373328 13306 tablet_bootstrap.cc:492] T a16acc33ab6c45858c52cceb4355207a P 2145d15ab2d74c0287839266f1411a43: Bootstrap complete.
I20240503 13:59:54.373281 9210 raft_consensus.cc:2932] T e92eaea6e9c8493690b749f162949ac4 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.373471 13306 ts_tablet_manager.cc:1367] T a16acc33ab6c45858c52cceb4355207a P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 13:59:54.374673 13306 raft_consensus.cc:348] T a16acc33ab6c45858c52cceb4355207a P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:54.374800 13306 raft_consensus.cc:729] T a16acc33ab6c45858c52cceb4355207a P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: LEARNER
I20240503 13:59:54.374897 13306 consensus_queue.cc:260] T a16acc33ab6c45858c52cceb4355207a P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:54.375509 13323 raft_consensus.cc:2229] T e2d398cb36d54bc785648751b5b52370 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:54.375432 6526 raft_consensus.cc:2932] T e92eaea6e9c8493690b749f162949ac4 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.375727 13323 raft_consensus.cc:2258] T e2d398cb36d54bc785648751b5b52370 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:54.376286 13323 ts_tablet_manager.cc:1875] T e2d398cb36d54bc785648751b5b52370 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:54.378466 6387 catalog_manager.cc:5484] T e92eaea6e9c8493690b749f162949ac4 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.379480 13306 ts_tablet_manager.cc:1398] T a16acc33ab6c45858c52cceb4355207a P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:59:54.380064 8968 tablet_copy_service.cc:342] P 06ca8dbc004c42b99faa527a4e6f9545: Request end of tablet copy session 2145d15ab2d74c0287839266f1411a43-a16acc33ab6c45858c52cceb4355207a received from {username='slave'} at 127.0.99.130:39847
I20240503 13:59:54.380144 8968 tablet_copy_service.cc:433] P 06ca8dbc004c42b99faa527a4e6f9545: ending tablet copy session 2145d15ab2d74c0287839266f1411a43-a16acc33ab6c45858c52cceb4355207a on tablet a16acc33ab6c45858c52cceb4355207a with peer 2145d15ab2d74c0287839266f1411a43
I20240503 13:59:54.380765 13323 ts_tablet_manager.cc:1888] T e2d398cb36d54bc785648751b5b52370 P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:54.380826 13323 log.cc:1196] T e2d398cb36d54bc785648751b5b52370 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/e2d398cb36d54bc785648751b5b52370
I20240503 13:59:54.381336 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet e2d398cb36d54bc785648751b5b52370 (table rebalancer_extra_table_72 [id=fb05f2e903bb419db0bfe7d54f35b083]) successfully deleted
I20240503 13:59:54.383105 8946 raft_consensus.cc:2932] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.387748 6657 consensus_queue.cc:237] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:54.389693 9210 raft_consensus.cc:1261] T d588a730a0c94d4f9446049e537c6727 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:54.389703 9081 raft_consensus.cc:1261] T d588a730a0c94d4f9446049e537c6727 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:54.390091 13325 consensus_queue.cc:1035] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:54.390195 11378 raft_consensus.cc:1050] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6: attempting to promote NON_VOTER 2145d15ab2d74c0287839266f1411a43 to VOTER
I20240503 13:59:54.390560 11378 consensus_queue.cc:237] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 13:59:54.391592 13325 consensus_queue.cc:1035] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:54.391938 13325 raft_consensus.cc:2932] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:54.392400 6922 raft_consensus.cc:1261] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.392771 13080 consensus_queue.cc:1035] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 13:59:54.392983 6595 consensus_peers.cc:485] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 -> Peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): Couldn't send request to peer f58065db70b64ef8b1dc499aa6ae4460. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d588a730a0c94d4f9446049e537c6727. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:54.393193 6657 raft_consensus.cc:1261] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.395215 13080 consensus_queue.cc:1035] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.395799 9343 consensus_queue.cc:237] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 13:59:54.395829 9210 raft_consensus.cc:2932] T d588a730a0c94d4f9446049e537c6727 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:54.397060 6790 raft_consensus.cc:1261] T 442c955ab10f4ecbafdc9bf89d8fb39e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.397228 6526 raft_consensus.cc:1261] T e92eaea6e9c8493690b749f162949ac4 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.397506 13242 consensus_queue.cc:1035] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.397583 8946 raft_consensus.cc:1261] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.398093 11167 consensus_queue.cc:1035] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.398423 11167 consensus_queue.cc:1035] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.399454 9081 raft_consensus.cc:2932] T d588a730a0c94d4f9446049e537c6727 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:54.400687 13080 raft_consensus.cc:2932] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.401005 6526 raft_consensus.cc:2932] T e92eaea6e9c8493690b749f162949ac4 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.402415 6381 catalog_manager.cc:5484] T d588a730a0c94d4f9446049e537c6727 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:54.405238 6374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e92eaea6e9c8493690b749f162949ac4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:54.405246 13242 raft_consensus.cc:2932] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:54.405503 8946 raft_consensus.cc:2932] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.406284 6384 catalog_manager.cc:5484] T e92eaea6e9c8493690b749f162949ac4 P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:54.407407 6790 raft_consensus.cc:2932] T 442c955ab10f4ecbafdc9bf89d8fb39e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:54.407502 6922 raft_consensus.cc:2932] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:54.411204 6386 catalog_manager.cc:5484] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:54.412655 6657 raft_consensus.cc:2932] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:54.414654 9192 tablet_service.cc:1514] Processing DeleteTablet for tablet e92eaea6e9c8493690b749f162949ac4 with delete_type TABLET_DATA_TOMBSTONED (TS ab1d21935d6049888eb0ceccc3c9d735 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57960
I20240503 13:59:54.414891 13295 tablet_replica.cc:330] T e92eaea6e9c8493690b749f162949ac4 P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 13:59:54.415050 13295 raft_consensus.cc:2229] T e92eaea6e9c8493690b749f162949ac4 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:54.415215 13295 raft_consensus.cc:2258] T e92eaea6e9c8493690b749f162949ac4 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:54.416348 13295 ts_tablet_manager.cc:1875] T e92eaea6e9c8493690b749f162949ac4 P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:54.418380 13295 ts_tablet_manager.cc:1888] T e92eaea6e9c8493690b749f162949ac4 P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:54.418432 13295 log.cc:1196] T e92eaea6e9c8493690b749f162949ac4 P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/e92eaea6e9c8493690b749f162949ac4
I20240503 13:59:54.418859 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet e92eaea6e9c8493690b749f162949ac4 (table rebalancer_extra_table_8 [id=9e8aad8638e34703aac55865282ca799]) successfully deleted
I20240503 13:59:54.421880 9343 consensus_queue.cc:237] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 13:59:54.423053 6922 raft_consensus.cc:1261] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.423058 6657 raft_consensus.cc:1261] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.423399 13242 consensus_queue.cc:1035] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.423555 13080 consensus_queue.cc:1035] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.425840 13242 raft_consensus.cc:2932] T 442c955ab10f4ecbafdc9bf89d8fb39e P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:54.426528 6657 raft_consensus.cc:2932] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:54.427393 6374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 442c955ab10f4ecbafdc9bf89d8fb39e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:54.427387 6922 raft_consensus.cc:2932] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:54.429718 6387 catalog_manager.cc:5484] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index 3 to 4, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:54.432225 6769 tablet_service.cc:1514] Processing DeleteTablet for tablet 442c955ab10f4ecbafdc9bf89d8fb39e with delete_type TABLET_DATA_TOMBSTONED (TS 4a85e492c8154e9881a9cc2a105d7e64 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:54.432480 13323 tablet_replica.cc:330] T 442c955ab10f4ecbafdc9bf89d8fb39e P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:54.432636 13323 raft_consensus.cc:2229] T 442c955ab10f4ecbafdc9bf89d8fb39e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:54.432716 9343 consensus_queue.cc:237] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:54.432817 13323 raft_consensus.cc:2258] T 442c955ab10f4ecbafdc9bf89d8fb39e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:54.433389 13323 ts_tablet_manager.cc:1875] T 442c955ab10f4ecbafdc9bf89d8fb39e P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 13:59:54.434479 9278 consensus_peers.cc:485] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 -> Peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): Couldn't send request to peer f58065db70b64ef8b1dc499aa6ae4460. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b397d0460c94438a773be184312b60f. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:54.435577 8946 raft_consensus.cc:1261] T 0b397d0460c94438a773be184312b60f P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:54.435638 13323 ts_tablet_manager.cc:1888] T 442c955ab10f4ecbafdc9bf89d8fb39e P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:54.435712 13323 log.cc:1196] T 442c955ab10f4ecbafdc9bf89d8fb39e P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/442c955ab10f4ecbafdc9bf89d8fb39e
I20240503 13:59:54.435982 9081 raft_consensus.cc:1261] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:54.436408 11378 consensus_queue.cc:1035] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:54.436542 11167 consensus_queue.cc:1035] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:54.436421 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet 442c955ab10f4ecbafdc9bf89d8fb39e (table rebalancer_extra_table_66 [id=7af8efee9ffd469b98e12ab7f8005bbc]) successfully deleted
I20240503 13:59:54.439263 13080 raft_consensus.cc:2932] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:54.439713 8947 raft_consensus.cc:2932] T 0b397d0460c94438a773be184312b60f P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:54.439761 9081 raft_consensus.cc:2932] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:54.443068 6381 catalog_manager.cc:5484] T 0b397d0460c94438a773be184312b60f P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:54.460594 9343 consensus_queue.cc:237] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:54.462196 9210 raft_consensus.cc:1261] T 6ee85f4483004c1f9499203c7323706d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:54.462554 11167 consensus_queue.cc:1035] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:54.462764 6922 raft_consensus.cc:1261] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:54.463198 11167 consensus_queue.cc:1035] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 13:59:54.464418 9279 consensus_peers.cc:485] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 -> Peer 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): Couldn't send request to peer 2145d15ab2d74c0287839266f1411a43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ee85f4483004c1f9499203c7323706d. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:54.465276 11167 raft_consensus.cc:2932] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:54.466257 6922 raft_consensus.cc:2932] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:54.469619 6381 catalog_manager.cc:5484] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:54.471051 9210 raft_consensus.cc:2932] T 6ee85f4483004c1f9499203c7323706d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:54.479338 13280 raft_consensus.cc:1050] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4: attempting to promote NON_VOTER 0d77752181e944698e809e325e5937e6 to VOTER
I20240503 13:59:54.479815 13280 consensus_queue.cc:237] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 13:59:54.481178 9343 raft_consensus.cc:1261] T 1ab63490264c4755aeab897f865e9a11 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.481498 6922 raft_consensus.cc:1261] T 1ab63490264c4755aeab897f865e9a11 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.481772 8947 raft_consensus.cc:1261] T 1ab63490264c4755aeab897f865e9a11 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.481949 12864 consensus_queue.cc:1035] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.482018 13300 raft_consensus.cc:1050] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64: attempting to promote NON_VOTER 5f46e19644fd4f4b825c86ba6af768da to VOTER
I20240503 13:59:54.482409 13300 consensus_queue.cc:237] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 13:59:54.482481 13158 consensus_queue.cc:1035] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.482615 12864 consensus_queue.cc:1035] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.483865 8946 raft_consensus.cc:1261] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.484337 6526 raft_consensus.cc:1261] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.484695 13116 consensus_queue.cc:1035] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.485236 13276 consensus_queue.cc:1035] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.485529 12748 raft_consensus.cc:1050] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460: attempting to promote NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 to VOTER
I20240503 13:59:54.485930 9343 raft_consensus.cc:1261] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.485927 12748 consensus_queue.cc:237] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 13:59:54.486280 13276 consensus_queue.cc:1035] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.487803 6659 raft_consensus.cc:1261] T af8bba02684c4555893def9c1e88ebf0 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.487957 6790 raft_consensus.cc:1261] T af8bba02684c4555893def9c1e88ebf0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.488193 11311 consensus_queue.cc:1035] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.488346 12748 consensus_queue.cc:1035] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.488538 9081 raft_consensus.cc:1261] T af8bba02684c4555893def9c1e88ebf0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.489161 11311 consensus_queue.cc:1035] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.492395 12748 raft_consensus.cc:2932] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:54.492883 6788 raft_consensus.cc:2932] T af8bba02684c4555893def9c1e88ebf0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:54.495137 9081 raft_consensus.cc:2932] T af8bba02684c4555893def9c1e88ebf0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:54.495930 6387 catalog_manager.cc:5484] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:54.497265 13276 raft_consensus.cc:2932] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.498526 8946 raft_consensus.cc:2932] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.499729 6526 raft_consensus.cc:2932] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.500295 9343 raft_consensus.cc:2932] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.502048 6659 raft_consensus.cc:2932] T af8bba02684c4555893def9c1e88ebf0 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:54.502573 6386 catalog_manager.cc:5484] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.506706 12748 raft_consensus.cc:1050] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460: attempting to promote NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 to VOTER
I20240503 13:59:54.507098 12748 consensus_queue.cc:237] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 13:59:54.507156 9079 consensus_queue.cc:237] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 13:59:54.507674 6922 consensus_queue.cc:237] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 13:59:54.508829 6788 raft_consensus.cc:1261] T af8bba02684c4555893def9c1e88ebf0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.509205 13159 consensus_queue.cc:1035] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.509398 9342 raft_consensus.cc:1261] T f1291c3100bb4533891a1a7a3357d363 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:54.509677 6657 raft_consensus.cc:1261] T af8bba02684c4555893def9c1e88ebf0 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.510430 13158 consensus_queue.cc:1035] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:54.510645 6659 raft_consensus.cc:1261] T cf7323c2594e4cd5967c3d38e4120815 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:54.511132 13159 consensus_queue.cc:1035] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.511523 13159 consensus_queue.cc:1035] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:54.512252 6527 raft_consensus.cc:1261] T f1291c3100bb4533891a1a7a3357d363 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:54.512516 6790 consensus_queue.cc:237] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
W20240503 13:59:54.512945 9016 consensus_peers.cc:485] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): Couldn't send request to peer 06ca8dbc004c42b99faa527a4e6f9545. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1291c3100bb4533891a1a7a3357d363. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:54.513514 13158 consensus_queue.cc:1035] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:54.513550 13280 raft_consensus.cc:2932] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:54.513980 8946 raft_consensus.cc:1261] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.514333 12032 consensus_queue.cc:1035] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.514410 6527 raft_consensus.cc:1261] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.515995 12032 consensus_queue.cc:1035] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.516597 9079 raft_consensus.cc:1261] T cf7323c2594e4cd5967c3d38e4120815 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:54.516815 13300 raft_consensus.cc:2932] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.517323 8946 raft_consensus.cc:2932] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.518036 6372 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c8b2d8a9ed5f4b09812a85445a8e3b85 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:54.519680 9343 raft_consensus.cc:2932] T 1ab63490264c4755aeab897f865e9a11 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:54.520061 11311 consensus_queue.cc:1035] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:54.520292 6385 catalog_manager.cc:5484] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:54.520335 6381 catalog_manager.cc:5484] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:54.520668 11311 raft_consensus.cc:2932] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:54.522742 6527 raft_consensus.cc:2932] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.523346 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet af8bba02684c4555893def9c1e88ebf0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:54.523969 6384 catalog_manager.cc:5484] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:54.526329 6659 raft_consensus.cc:2932] T af8bba02684c4555893def9c1e88ebf0 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:54.526619 6526 raft_consensus.cc:1261] T cf7323c2594e4cd5967c3d38e4120815 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:54.527086 13370 consensus_queue.cc:1035] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:54.528774 9321 tablet_service.cc:1514] Processing DeleteTablet for tablet c8b2d8a9ed5f4b09812a85445a8e3b85 with delete_type TABLET_DATA_TOMBSTONED (TS 0d77752181e944698e809e325e5937e6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:54.528992 13294 tablet_replica.cc:330] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:54.529147 13294 raft_consensus.cc:2229] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:54.529284 13294 raft_consensus.cc:2258] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:54.530063 13294 ts_tablet_manager.cc:1875] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:54.531392 9060 tablet_service.cc:1514] Processing DeleteTablet for tablet af8bba02684c4555893def9c1e88ebf0 with delete_type TABLET_DATA_TOMBSTONED (TS 8d574fb4a89d4eea947c859470d44dd4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:54.531700 9081 consensus_queue.cc:237] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 13:59:54.532248 13294 ts_tablet_manager.cc:1888] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:54.532317 13294 log.cc:1196] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/c8b2d8a9ed5f4b09812a85445a8e3b85
I20240503 13:59:54.532650 8947 raft_consensus.cc:2932] T 1ab63490264c4755aeab897f865e9a11 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:54.532828 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet c8b2d8a9ed5f4b09812a85445a8e3b85 (table rebalancer_extra_table_28 [id=0d58a21c125546e9acb619279319e0f5]) successfully deleted
I20240503 13:59:54.533465 6922 raft_consensus.cc:2932] T 1ab63490264c4755aeab897f865e9a11 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:54.535118 13373 tablet_replica.cc:330] T af8bba02684c4555893def9c1e88ebf0 P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:54.535269 13373 raft_consensus.cc:2229] T af8bba02684c4555893def9c1e88ebf0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:54.535478 13373 raft_consensus.cc:2258] T af8bba02684c4555893def9c1e88ebf0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:54.537065 13373 ts_tablet_manager.cc:1875] T af8bba02684c4555893def9c1e88ebf0 P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:54.537101 9342 raft_consensus.cc:1261] T 1ab63490264c4755aeab897f865e9a11 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.537597 6922 raft_consensus.cc:1261] T 1ab63490264c4755aeab897f865e9a11 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.538093 12864 consensus_queue.cc:1035] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.538244 12372 consensus_queue.cc:1035] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.540830 13373 ts_tablet_manager.cc:1888] T af8bba02684c4555893def9c1e88ebf0 P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:54.541075 6790 raft_consensus.cc:2932] T af8bba02684c4555893def9c1e88ebf0 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:54.541617 13373 log.cc:1196] T af8bba02684c4555893def9c1e88ebf0 P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/af8bba02684c4555893def9c1e88ebf0
I20240503 13:59:54.541790 12372 raft_consensus.cc:2932] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:54.543980 12748 raft_consensus.cc:2932] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.546170 6527 raft_consensus.cc:2932] T cf7323c2594e4cd5967c3d38e4120815 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.546366 6659 raft_consensus.cc:2932] T cf7323c2594e4cd5967c3d38e4120815 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.546900 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet af8bba02684c4555893def9c1e88ebf0 (table rebalancer_extra_table_70 [id=0d758a66160e4c21a4e6c8f5b3cb6179]) successfully deleted
I20240503 13:59:54.547286 6526 raft_consensus.cc:2932] T f1291c3100bb4533891a1a7a3357d363 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:54.549285 6379 catalog_manager.cc:5484] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index -1 to 2, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:54.550144 9342 raft_consensus.cc:2932] T f1291c3100bb4533891a1a7a3357d363 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:54.550666 12864 raft_consensus.cc:2932] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:54.551223 6387 catalog_manager.cc:5484] T cf7323c2594e4cd5967c3d38e4120815 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index 4 to 5, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.552636 9081 raft_consensus.cc:2932] T cf7323c2594e4cd5967c3d38e4120815 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.554646 9342 raft_consensus.cc:2932] T 1ab63490264c4755aeab897f865e9a11 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:54.556915 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1ab63490264c4755aeab897f865e9a11 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:54.558409 6922 raft_consensus.cc:2932] T 1ab63490264c4755aeab897f865e9a11 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:54.561933 6381 catalog_manager.cc:5484] T 1ab63490264c4755aeab897f865e9a11 P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:54.564810 8926 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab63490264c4755aeab897f865e9a11 with delete_type TABLET_DATA_TOMBSTONED (TS 06ca8dbc004c42b99faa527a4e6f9545 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:54.565310 6922 consensus_queue.cc:237] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 13:59:54.565582 6526 consensus_queue.cc:237] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:54.565933 13164 tablet_replica.cc:330] T 1ab63490264c4755aeab897f865e9a11 P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:54.566083 13164 raft_consensus.cc:2229] T 1ab63490264c4755aeab897f865e9a11 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:54.566221 13164 raft_consensus.cc:2258] T 1ab63490264c4755aeab897f865e9a11 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:54.566504 9081 raft_consensus.cc:1261] T cf7323c2594e4cd5967c3d38e4120815 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:54.566843 13164 ts_tablet_manager.cc:1875] T 1ab63490264c4755aeab897f865e9a11 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:54.567118 6527 raft_consensus.cc:1261] T cf7323c2594e4cd5967c3d38e4120815 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:54.567595 13365 consensus_queue.cc:1035] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:54.567945 13365 consensus_queue.cc:1035] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:54.569852 12748 raft_consensus.cc:2932] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.570039 13164 ts_tablet_manager.cc:1888] T 1ab63490264c4755aeab897f865e9a11 P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:54.570173 13164 log.cc:1196] T 1ab63490264c4755aeab897f865e9a11 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/1ab63490264c4755aeab897f865e9a11
I20240503 13:59:54.570807 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 1ab63490264c4755aeab897f865e9a11 (table rebalancer_extra_table_54 [id=0179e5ae38524fc9b8f37ce24d941401]) successfully deleted
I20240503 13:59:54.571727 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cf7323c2594e4cd5967c3d38e4120815 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:54.572508 9342 raft_consensus.cc:1261] T aecb348d4b3b4500a587c92a7fefab4e P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:54.572907 13289 consensus_queue.cc:1035] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:54.574599 9079 raft_consensus.cc:2932] T cf7323c2594e4cd5967c3d38e4120815 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.572631 9081 raft_consensus.cc:1261] T aecb348d4b3b4500a587c92a7fefab4e P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:54.575721 6464 consensus_peers.cc:485] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da -> Peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): Couldn't send request to peer f58065db70b64ef8b1dc499aa6ae4460. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aecb348d4b3b4500a587c92a7fefab4e. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:54.575953 13289 consensus_queue.cc:1035] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:54.576321 6526 raft_consensus.cc:2932] T cf7323c2594e4cd5967c3d38e4120815 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:54.577230 6379 catalog_manager.cc:5484] T cf7323c2594e4cd5967c3d38e4120815 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:54.580468 6639 tablet_service.cc:1514] Processing DeleteTablet for tablet cf7323c2594e4cd5967c3d38e4120815 with delete_type TABLET_DATA_TOMBSTONED (TS 2145d15ab2d74c0287839266f1411a43 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:54.580727 13315 tablet_replica.cc:330] T cf7323c2594e4cd5967c3d38e4120815 P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:54.580891 13315 raft_consensus.cc:2229] T cf7323c2594e4cd5967c3d38e4120815 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:54.580911 13289 raft_consensus.cc:2932] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:54.581079 13315 raft_consensus.cc:2258] T cf7323c2594e4cd5967c3d38e4120815 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:54.581909 9079 raft_consensus.cc:2932] T aecb348d4b3b4500a587c92a7fefab4e P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:54.583979 13315 ts_tablet_manager.cc:1875] T cf7323c2594e4cd5967c3d38e4120815 P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:54.584074 9343 raft_consensus.cc:2932] T aecb348d4b3b4500a587c92a7fefab4e P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:54.586200 13315 ts_tablet_manager.cc:1888] T cf7323c2594e4cd5967c3d38e4120815 P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:54.586272 13315 log.cc:1196] T cf7323c2594e4cd5967c3d38e4120815 P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/cf7323c2594e4cd5967c3d38e4120815
I20240503 13:59:54.586839 6385 catalog_manager.cc:5484] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:54.587430 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet cf7323c2594e4cd5967c3d38e4120815 (table rebalance_test_table_11 [id=3e15c9008ce84692a697ae9651a18574]) successfully deleted
I20240503 13:59:54.650755 6790 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "64135eb6ef574c95b3677e32d66748a4"
dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57832
I20240503 13:59:54.650913 6790 raft_consensus.cc:595] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:54.698997 13300 raft_consensus.cc:979] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64: : Instructing follower 8d574fb4a89d4eea947c859470d44dd4 to start an election
I20240503 13:59:54.699136 13300 raft_consensus.cc:1067] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Signalling peer 8d574fb4a89d4eea947c859470d44dd4 to start an election
I20240503 13:59:54.700174 9079 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "64135eb6ef574c95b3677e32d66748a4"
dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
from {username='slave'} at 127.0.99.131:44581
I20240503 13:59:54.700367 9079 raft_consensus.cc:482] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:54.700448 9079 raft_consensus.cc:3037] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:54.701447 9079 raft_consensus.cc:504] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 13:59:54.701925 9079 leader_election.cc:288] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:54.702445 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64135eb6ef574c95b3677e32d66748a4" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 13:59:54.702433 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64135eb6ef574c95b3677e32d66748a4" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0d77752181e944698e809e325e5937e6"
I20240503 13:59:54.702562 9343 raft_consensus.cc:3037] T 64135eb6ef574c95b3677e32d66748a4 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:54.702576 6922 raft_consensus.cc:3037] T 64135eb6ef574c95b3677e32d66748a4 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:54.703617 6922 raft_consensus.cc:2454] T 64135eb6ef574c95b3677e32d66748a4 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 2.
I20240503 13:59:54.703886 9343 raft_consensus.cc:2454] T 64135eb6ef574c95b3677e32d66748a4 P 0d77752181e944698e809e325e5937e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 2.
I20240503 13:59:54.704234 9017 leader_election.cc:302] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0d77752181e944698e809e325e5937e6, 8d574fb4a89d4eea947c859470d44dd4, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 13:59:54.704419 13375 raft_consensus.cc:2781] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:54.704520 13375 raft_consensus.cc:686] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 LEADER]: Becoming Leader. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 13:59:54.704666 13375 consensus_queue.cc:237] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 13:59:54.704998 12063 raft_consensus.cc:1050] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735: attempting to promote NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 to VOTER
I20240503 13:59:54.705410 12063 consensus_queue.cc:237] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
I20240503 13:59:54.707108 9342 raft_consensus.cc:1261] T db13f399b9e04fe9a69e729e384d046c P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.707154 6788 raft_consensus.cc:1261] T db13f399b9e04fe9a69e729e384d046c P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.707371 6384 catalog_manager.cc:5484] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: term changed from 1 to 2, leader changed from 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) to 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134). New cstate: current_term: 2 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:54.707630 12063 consensus_queue.cc:1035] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.708812 11471 consensus_queue.cc:1035] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.708832 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64135eb6ef574c95b3677e32d66748a4" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
I20240503 13:59:54.708927 6790 raft_consensus.cc:3032] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:54.708966 6790 raft_consensus.cc:729] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:54.709120 6790 consensus_queue.cc:260] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 13:59:54.709326 6790 raft_consensus.cc:3037] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:54.710294 6790 raft_consensus.cc:2454] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 2.
I20240503 13:59:54.710633 8947 raft_consensus.cc:1261] T db13f399b9e04fe9a69e729e384d046c P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.711395 11471 consensus_queue.cc:1035] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.713428 13484 raft_consensus.cc:2932] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:54.714176 9342 raft_consensus.cc:2932] T db13f399b9e04fe9a69e729e384d046c P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:54.716040 8947 raft_consensus.cc:2932] T db13f399b9e04fe9a69e729e384d046c P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:54.718659 6379 catalog_manager.cc:5484] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index 2 to 3, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:54.720835 6791 raft_consensus.cc:2932] T db13f399b9e04fe9a69e729e384d046c P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:54.724290 9210 consensus_queue.cc:237] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
I20240503 13:59:54.726190 8947 raft_consensus.cc:1261] T db13f399b9e04fe9a69e729e384d046c P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.726375 9342 raft_consensus.cc:1261] T db13f399b9e04fe9a69e729e384d046c P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.726753 13481 consensus_queue.cc:1035] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.727072 13481 consensus_queue.cc:1035] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.728865 13481 raft_consensus.cc:2932] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:54.729301 8947 raft_consensus.cc:2932] T db13f399b9e04fe9a69e729e384d046c P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:54.729530 9342 raft_consensus.cc:2932] T db13f399b9e04fe9a69e729e384d046c P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:54.730614 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet db13f399b9e04fe9a69e729e384d046c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:54.732658 6379 catalog_manager.cc:5484] T db13f399b9e04fe9a69e729e384d046c P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index 3 to 4, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:54.740092 6769 tablet_service.cc:1514] Processing DeleteTablet for tablet db13f399b9e04fe9a69e729e384d046c with delete_type TABLET_DATA_TOMBSTONED (TS 4a85e492c8154e9881a9cc2a105d7e64 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:54.740459 13323 tablet_replica.cc:330] T db13f399b9e04fe9a69e729e384d046c P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:54.740621 13323 raft_consensus.cc:2229] T db13f399b9e04fe9a69e729e384d046c P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:54.740764 13323 raft_consensus.cc:2258] T db13f399b9e04fe9a69e729e384d046c P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:54.741415 13323 ts_tablet_manager.cc:1875] T db13f399b9e04fe9a69e729e384d046c P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:54.743744 13323 ts_tablet_manager.cc:1888] T db13f399b9e04fe9a69e729e384d046c P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:54.743824 13323 log.cc:1196] T db13f399b9e04fe9a69e729e384d046c P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/db13f399b9e04fe9a69e729e384d046c
I20240503 13:59:54.744323 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet db13f399b9e04fe9a69e729e384d046c (table rebalancer_extra_table_56 [id=bfd5a93b6dc141158909aa98a0ef7189]) successfully deleted
I20240503 13:59:54.747300 6922 consensus_queue.cc:237] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:54.748919 6659 raft_consensus.cc:1261] T 400c54bdecfa44e08c7eca046bcd2ffc P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.749365 13370 consensus_queue.cc:1035] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.749727 6526 raft_consensus.cc:1261] T 400c54bdecfa44e08c7eca046bcd2ffc P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:59:54.750171 6857 consensus_peers.cc:485] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): Couldn't send request to peer 8d574fb4a89d4eea947c859470d44dd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 400c54bdecfa44e08c7eca046bcd2ffc. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:54.750662 13370 consensus_queue.cc:1035] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.751080 13300 raft_consensus.cc:1050] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64: attempting to promote NON_VOTER 5f46e19644fd4f4b825c86ba6af768da to VOTER
I20240503 13:59:54.751446 13300 consensus_queue.cc:237] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 13:59:54.752621 13159 raft_consensus.cc:2932] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:54.752746 9210 raft_consensus.cc:1261] T 957b89de328d434d8ea03a59bc93fdcb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.752831 8947 raft_consensus.cc:1261] T 957b89de328d434d8ea03a59bc93fdcb P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.753159 12528 consensus_queue.cc:1035] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.753818 12528 consensus_queue.cc:1035] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.754550 6527 raft_consensus.cc:1261] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.755151 12528 consensus_queue.cc:1035] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.757956 6381 catalog_manager.cc:5484] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index -1 to 4, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:54.758736 13116 raft_consensus.cc:2932] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.759991 6526 raft_consensus.cc:2932] T 400c54bdecfa44e08c7eca046bcd2ffc P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:54.762014 6659 raft_consensus.cc:2932] T 400c54bdecfa44e08c7eca046bcd2ffc P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:54.764050 6381 catalog_manager.cc:5484] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:54.766383 8947 raft_consensus.cc:2932] T 957b89de328d434d8ea03a59bc93fdcb P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.767179 6791 consensus_queue.cc:237] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 13:59:54.770452 8947 raft_consensus.cc:1261] T 957b89de328d434d8ea03a59bc93fdcb P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.770905 13276 consensus_queue.cc:1035] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.768855 6526 raft_consensus.cc:2932] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.772958 12528 raft_consensus.cc:2932] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.773491 6528 raft_consensus.cc:1261] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.774431 6372 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 957b89de328d434d8ea03a59bc93fdcb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:54.776417 6381 catalog_manager.cc:5484] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:54.778455 6526 consensus_queue.cc:237] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:54.779356 9192 tablet_service.cc:1514] Processing DeleteTablet for tablet 957b89de328d434d8ea03a59bc93fdcb with delete_type TABLET_DATA_TOMBSTONED (TS ab1d21935d6049888eb0ceccc3c9d735 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57960
I20240503 13:59:54.779579 13295 tablet_replica.cc:330] T 957b89de328d434d8ea03a59bc93fdcb P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 13:59:54.779747 13295 raft_consensus.cc:2229] T 957b89de328d434d8ea03a59bc93fdcb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:54.779865 13295 pending_rounds.cc:62] T 957b89de328d434d8ea03a59bc93fdcb P ab1d21935d6049888eb0ceccc3c9d735: Trying to abort 1 pending ops.
I20240503 13:59:54.779820 8947 raft_consensus.cc:2932] T 957b89de328d434d8ea03a59bc93fdcb P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.779924 13295 pending_rounds.cc:68] T 957b89de328d434d8ea03a59bc93fdcb P ab1d21935d6049888eb0ceccc3c9d735: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023594679300730880 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "957b89de328d434d8ea03a59bc93fdcb" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } } }
I20240503 13:59:54.780092 13295 raft_consensus.cc:2894] T 957b89de328d434d8ea03a59bc93fdcb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240503 13:59:54.780174 13295 raft_consensus.cc:2866] T 957b89de328d434d8ea03a59bc93fdcb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 13:59:54.780218 13295 raft_consensus.cc:2258] T 957b89de328d434d8ea03a59bc93fdcb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:54.780812 13295 ts_tablet_manager.cc:1875] T 957b89de328d434d8ea03a59bc93fdcb P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:54.781245 13276 consensus_queue.cc:1035] T 957b89de328d434d8ea03a59bc93fdcb P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.782493 6659 raft_consensus.cc:1261] T e106b1432baf423493c95fcce63a55da P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:59:54.782732 6465 consensus_peers.cc:485] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e106b1432baf423493c95fcce63a55da. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:54.782887 13295 ts_tablet_manager.cc:1888] T 957b89de328d434d8ea03a59bc93fdcb P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:54.782948 13295 log.cc:1196] T 957b89de328d434d8ea03a59bc93fdcb P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/957b89de328d434d8ea03a59bc93fdcb
I20240503 13:59:54.782918 13157 consensus_queue.cc:1035] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.783435 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet 957b89de328d434d8ea03a59bc93fdcb (table rebalancer_extra_table_20 [id=c8e9e03cbc904134b1bfd2851cf3b5a7]) successfully deleted
I20240503 13:59:54.783632 6528 raft_consensus.cc:2932] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:54.786192 6791 raft_consensus.cc:1261] T e106b1432baf423493c95fcce63a55da P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.786584 13157 consensus_queue.cc:1035] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.788548 13289 raft_consensus.cc:2932] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:54.788908 6791 raft_consensus.cc:2932] T e106b1432baf423493c95fcce63a55da P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:54.791254 6659 raft_consensus.cc:2932] T e106b1432baf423493c95fcce63a55da P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:54.791684 6381 catalog_manager.cc:5484] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:54.799242 6659 raft_consensus.cc:1203] T a16acc33ab6c45858c52cceb4355207a P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:54.803515 6528 consensus_queue.cc:237] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:54.804914 6922 raft_consensus.cc:1261] T da69140936424a31b66c87e2ad4092cb P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:59:54.805202 6464 consensus_peers.cc:485] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da -> Peer ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): Couldn't send request to peer ab1d21935d6049888eb0ceccc3c9d735. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da69140936424a31b66c87e2ad4092cb. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:54.805259 6659 raft_consensus.cc:1261] T da69140936424a31b66c87e2ad4092cb P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.805414 13157 consensus_queue.cc:1035] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.805671 12156 consensus_queue.cc:1035] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.807158 12156 raft_consensus.cc:2932] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:54.807605 6922 raft_consensus.cc:2932] T da69140936424a31b66c87e2ad4092cb P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:54.808075 6659 raft_consensus.cc:2932] T da69140936424a31b66c87e2ad4092cb P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:54.811257 6379 catalog_manager.cc:5484] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index -1 to 4, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:54.827778 6791 consensus_queue.cc:237] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:54.829931 6528 raft_consensus.cc:1261] T 01551845458048e48df9901947b4eec6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:54.830802 8947 raft_consensus.cc:1261] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:54.831480 13116 consensus_queue.cc:1035] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:54.831787 13300 consensus_queue.cc:1035] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 13:59:54.833053 6726 consensus_peers.cc:485] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 -> Peer 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): Couldn't send request to peer 8d574fb4a89d4eea947c859470d44dd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01551845458048e48df9901947b4eec6. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:54.834805 13300 raft_consensus.cc:2932] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:54.835134 8947 raft_consensus.cc:2932] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:54.838456 6381 catalog_manager.cc:5484] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index -1 to 2, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:54.841974 6528 raft_consensus.cc:2932] T 01551845458048e48df9901947b4eec6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:54.852421 9342 consensus_queue.cc:237] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:54.853709 6922 raft_consensus.cc:1261] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:54.853888 8947 raft_consensus.cc:1261] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:54.854152 13080 consensus_queue.cc:1035] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:54.854312 13242 consensus_queue.cc:1035] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 13:59:54.854442 9277 consensus_peers.cc:485] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a25af1e7ae64f4fad8ad82faa1c6eb6. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:54.855358 13540 ts_tablet_manager.cc:902] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460: Initiating tablet copy from peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:54.855974 13540 tablet_copy_client.cc:308] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:41931
I20240503 13:59:54.856487 11378 raft_consensus.cc:2932] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:54.856981 9362 tablet_copy_service.cc:140] P 0d77752181e944698e809e325e5937e6: Received BeginTabletCopySession request for tablet 0b397d0460c94438a773be184312b60f from peer f58065db70b64ef8b1dc499aa6ae4460 ({username='slave'} at 127.0.99.132:60419)
I20240503 13:59:54.856904 8947 raft_consensus.cc:2932] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:54.857071 9362 tablet_copy_service.cc:161] P 0d77752181e944698e809e325e5937e6: Beginning new tablet copy session on tablet 0b397d0460c94438a773be184312b60f from peer f58065db70b64ef8b1dc499aa6ae4460 at {username='slave'} at 127.0.99.132:60419: session id = f58065db70b64ef8b1dc499aa6ae4460-0b397d0460c94438a773be184312b60f
I20240503 13:59:54.858067 9362 tablet_copy_source_session.cc:205] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:54.860416 6381 catalog_manager.cc:5484] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:54.862174 13540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b397d0460c94438a773be184312b60f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:54.862346 6922 raft_consensus.cc:2932] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:54.864413 13540 tablet_copy_client.cc:783] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:54.864547 13540 tablet_copy_client.cc:647] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:54.865896 13540 tablet_copy_client.cc:523] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:54.866969 13540 tablet_bootstrap.cc:492] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:54.870997 13540 tablet_bootstrap.cc:492] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:54.871263 13540 tablet_bootstrap.cc:492] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap complete.
I20240503 13:59:54.871443 13540 ts_tablet_manager.cc:1367] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240503 13:59:54.871865 13540 raft_consensus.cc:348] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:54.871985 13540 raft_consensus.cc:729] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: LEARNER
I20240503 13:59:54.872089 13540 consensus_queue.cc:260] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:54.872516 13540 ts_tablet_manager.cc:1398] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:54.872649 6791 consensus_queue.cc:237] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:54.873257 9362 tablet_copy_service.cc:342] P 0d77752181e944698e809e325e5937e6: Request end of tablet copy session f58065db70b64ef8b1dc499aa6ae4460-0b397d0460c94438a773be184312b60f received from {username='slave'} at 127.0.99.132:60419
I20240503 13:59:54.873335 9362 tablet_copy_service.cc:433] P 0d77752181e944698e809e325e5937e6: ending tablet copy session f58065db70b64ef8b1dc499aa6ae4460-0b397d0460c94438a773be184312b60f on tablet 0b397d0460c94438a773be184312b60f with peer f58065db70b64ef8b1dc499aa6ae4460
I20240503 13:59:54.874347 6657 raft_consensus.cc:1261] T ea45c42d7fed4171a1b726b0684d2fa5 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:59:54.874467 6727 consensus_peers.cc:485] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea45c42d7fed4171a1b726b0684d2fa5. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:54.874702 12528 consensus_queue.cc:1035] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.874783 6922 raft_consensus.cc:1261] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:54.875233 12528 consensus_queue.cc:1035] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.877038 12032 raft_consensus.cc:2932] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:54.878285 6922 raft_consensus.cc:2932] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:54.878989 6657 raft_consensus.cc:2932] T ea45c42d7fed4171a1b726b0684d2fa5 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:54.882439 6381 catalog_manager.cc:5484] T ea45c42d7fed4171a1b726b0684d2fa5 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:54.882808 13540 ts_tablet_manager.cc:902] T d588a730a0c94d4f9446049e537c6727 P f58065db70b64ef8b1dc499aa6ae4460: Initiating tablet copy from peer 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:54.883208 13540 tablet_copy_client.cc:308] T d588a730a0c94d4f9446049e537c6727 P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46119
I20240503 13:59:54.883985 6680 tablet_copy_service.cc:140] P 2145d15ab2d74c0287839266f1411a43: Received BeginTabletCopySession request for tablet d588a730a0c94d4f9446049e537c6727 from peer f58065db70b64ef8b1dc499aa6ae4460 ({username='slave'} at 127.0.99.132:45437)
I20240503 13:59:54.884083 6680 tablet_copy_service.cc:161] P 2145d15ab2d74c0287839266f1411a43: Beginning new tablet copy session on tablet d588a730a0c94d4f9446049e537c6727 from peer f58065db70b64ef8b1dc499aa6ae4460 at {username='slave'} at 127.0.99.132:45437: session id = f58065db70b64ef8b1dc499aa6ae4460-d588a730a0c94d4f9446049e537c6727
I20240503 13:59:54.885023 6680 tablet_copy_source_session.cc:205] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:54.885864 13540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d588a730a0c94d4f9446049e537c6727. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:54.887952 13540 tablet_copy_client.cc:783] T d588a730a0c94d4f9446049e537c6727 P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:54.888098 13540 tablet_copy_client.cc:647] T d588a730a0c94d4f9446049e537c6727 P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:54.889178 13540 tablet_copy_client.cc:523] T d588a730a0c94d4f9446049e537c6727 P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:54.890463 13540 tablet_bootstrap.cc:492] T d588a730a0c94d4f9446049e537c6727 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:54.894260 13540 tablet_bootstrap.cc:492] T d588a730a0c94d4f9446049e537c6727 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:54.894323 9081 consensus_queue.cc:237] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:54.894557 13540 tablet_bootstrap.cc:492] T d588a730a0c94d4f9446049e537c6727 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap complete.
I20240503 13:59:54.894712 13540 ts_tablet_manager.cc:1367] T d588a730a0c94d4f9446049e537c6727 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 13:59:54.895143 13540 raft_consensus.cc:348] T d588a730a0c94d4f9446049e537c6727 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:54.895255 13540 raft_consensus.cc:729] T d588a730a0c94d4f9446049e537c6727 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: LEARNER
I20240503 13:59:54.895399 13540 consensus_queue.cc:260] T d588a730a0c94d4f9446049e537c6727 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:54.895712 13540 ts_tablet_manager.cc:1398] T d588a730a0c94d4f9446049e537c6727 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 13:59:54.896051 9210 raft_consensus.cc:1261] T 44eddd53c3f348a9a1117a20473cbb65 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:54.896082 9016 consensus_peers.cc:485] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): Couldn't send request to peer 5f46e19644fd4f4b825c86ba6af768da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44eddd53c3f348a9a1117a20473cbb65. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:54.896572 6680 tablet_copy_service.cc:342] P 2145d15ab2d74c0287839266f1411a43: Request end of tablet copy session f58065db70b64ef8b1dc499aa6ae4460-d588a730a0c94d4f9446049e537c6727 received from {username='slave'} at 127.0.99.132:45437
I20240503 13:59:54.896656 6680 tablet_copy_service.cc:433] P 2145d15ab2d74c0287839266f1411a43: ending tablet copy session f58065db70b64ef8b1dc499aa6ae4460-d588a730a0c94d4f9446049e537c6727 on tablet d588a730a0c94d4f9446049e537c6727 with peer f58065db70b64ef8b1dc499aa6ae4460
I20240503 13:59:54.896679 12864 consensus_queue.cc:1035] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:54.897027 9342 raft_consensus.cc:1261] T 44eddd53c3f348a9a1117a20473cbb65 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:54.897778 12864 consensus_queue.cc:1035] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:54.898653 12864 raft_consensus.cc:2932] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:54.901042 9342 raft_consensus.cc:2932] T 44eddd53c3f348a9a1117a20473cbb65 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:54.902516 9205 raft_consensus.cc:2932] T 44eddd53c3f348a9a1117a20473cbb65 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:54.903534 6385 catalog_manager.cc:5484] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:54.913839 6657 consensus_queue.cc:237] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:54.915241 9081 raft_consensus.cc:1261] T 61dfb1d40ee444368470a0dafc3179a4 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:54.915364 6595 consensus_peers.cc:485] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 -> Peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): Couldn't send request to peer f58065db70b64ef8b1dc499aa6ae4460. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61dfb1d40ee444368470a0dafc3179a4. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:54.915566 9342 raft_consensus.cc:1261] T 61dfb1d40ee444368470a0dafc3179a4 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:54.915956 12915 consensus_queue.cc:1035] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:54.916121 13575 consensus_queue.cc:1035] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:54.917819 13325 raft_consensus.cc:2932] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:54.918191 9081 raft_consensus.cc:2932] T 61dfb1d40ee444368470a0dafc3179a4 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:54.919395 9342 raft_consensus.cc:2932] T 61dfb1d40ee444368470a0dafc3179a4 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:54.920748 6381 catalog_manager.cc:5484] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:54.937615 9342 consensus_queue.cc:237] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:54.938957 6657 raft_consensus.cc:1261] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:54.939069 8947 raft_consensus.cc:1261] T 4a1398d983c54a7498257f766052e8e9 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:54.939146 9277 consensus_peers.cc:485] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a1398d983c54a7498257f766052e8e9. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:54.939522 13080 consensus_queue.cc:1035] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:54.939724 13242 consensus_queue.cc:1035] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:54.941809 13242 raft_consensus.cc:2932] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:54.942584 8947 raft_consensus.cc:2932] T 4a1398d983c54a7498257f766052e8e9 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:54.945137 6657 raft_consensus.cc:2932] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:54.945241 6384 catalog_manager.cc:5484] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:54.958412 8947 consensus_queue.cc:237] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:54.959970 6528 raft_consensus.cc:1261] T ef19b1e5f15e49448c1b2919a24090d3 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:54.960358 13015 consensus_queue.cc:1035] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:54.960541 6657 raft_consensus.cc:1261] T ef19b1e5f15e49448c1b2919a24090d3 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:54.960563 8884 consensus_peers.cc:485] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef19b1e5f15e49448c1b2919a24090d3. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:54.961009 13015 consensus_queue.cc:1035] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:54.962144 11750 raft_consensus.cc:2932] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:54.962586 6528 raft_consensus.cc:2932] T ef19b1e5f15e49448c1b2919a24090d3 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:54.964460 6657 raft_consensus.cc:2932] T ef19b1e5f15e49448c1b2919a24090d3 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:54.965902 6381 catalog_manager.cc:5484] T ef19b1e5f15e49448c1b2919a24090d3 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index -1 to 2, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:54.982853 13604 ts_tablet_manager.cc:902] T f1291c3100bb4533891a1a7a3357d363 P 06ca8dbc004c42b99faa527a4e6f9545: Initiating tablet copy from peer 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:54.983440 13604 tablet_copy_client.cc:308] T f1291c3100bb4533891a1a7a3357d363 P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:39481
I20240503 13:59:54.983871 9101 tablet_copy_service.cc:140] P 8d574fb4a89d4eea947c859470d44dd4: Received BeginTabletCopySession request for tablet f1291c3100bb4533891a1a7a3357d363 from peer 06ca8dbc004c42b99faa527a4e6f9545 ({username='slave'} at 127.0.99.133:46917)
I20240503 13:59:54.983966 9101 tablet_copy_service.cc:161] P 8d574fb4a89d4eea947c859470d44dd4: Beginning new tablet copy session on tablet f1291c3100bb4533891a1a7a3357d363 from peer 06ca8dbc004c42b99faa527a4e6f9545 at {username='slave'} at 127.0.99.133:46917: session id = 06ca8dbc004c42b99faa527a4e6f9545-f1291c3100bb4533891a1a7a3357d363
I20240503 13:59:54.984831 9101 tablet_copy_source_session.cc:205] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:54.985810 13604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1291c3100bb4533891a1a7a3357d363. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:54.987670 13604 tablet_copy_client.cc:783] T f1291c3100bb4533891a1a7a3357d363 P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:54.987818 13604 tablet_copy_client.cc:647] T f1291c3100bb4533891a1a7a3357d363 P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:54.988886 13604 tablet_copy_client.cc:523] T f1291c3100bb4533891a1a7a3357d363 P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:54.990025 13604 tablet_bootstrap.cc:492] T f1291c3100bb4533891a1a7a3357d363 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:54.994350 13604 tablet_bootstrap.cc:492] T f1291c3100bb4533891a1a7a3357d363 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:54.994622 13604 tablet_bootstrap.cc:492] T f1291c3100bb4533891a1a7a3357d363 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap complete.
I20240503 13:59:54.994791 13604 ts_tablet_manager.cc:1367] T f1291c3100bb4533891a1a7a3357d363 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 13:59:54.995330 13604 raft_consensus.cc:348] T f1291c3100bb4533891a1a7a3357d363 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 13:59:54.995461 13604 raft_consensus.cc:729] T f1291c3100bb4533891a1a7a3357d363 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: LEARNER
I20240503 13:59:54.995570 13604 consensus_queue.cc:260] T f1291c3100bb4533891a1a7a3357d363 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 13:59:54.995867 13604 ts_tablet_manager.cc:1398] T f1291c3100bb4533891a1a7a3357d363 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 13:59:54.996317 9101 tablet_copy_service.cc:342] P 8d574fb4a89d4eea947c859470d44dd4: Request end of tablet copy session 06ca8dbc004c42b99faa527a4e6f9545-f1291c3100bb4533891a1a7a3357d363 received from {username='slave'} at 127.0.99.133:46917
I20240503 13:59:54.996395 9101 tablet_copy_service.cc:433] P 8d574fb4a89d4eea947c859470d44dd4: ending tablet copy session 06ca8dbc004c42b99faa527a4e6f9545-f1291c3100bb4533891a1a7a3357d363 on tablet f1291c3100bb4533891a1a7a3357d363 with peer 06ca8dbc004c42b99faa527a4e6f9545
I20240503 13:59:55.024631 13635 ts_tablet_manager.cc:902] T 6ee85f4483004c1f9499203c7323706d P 2145d15ab2d74c0287839266f1411a43: Initiating tablet copy from peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:55.025501 13635 tablet_copy_client.cc:308] T 6ee85f4483004c1f9499203c7323706d P 2145d15ab2d74c0287839266f1411a43: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:41931
I20240503 13:59:55.025921 9362 tablet_copy_service.cc:140] P 0d77752181e944698e809e325e5937e6: Received BeginTabletCopySession request for tablet 6ee85f4483004c1f9499203c7323706d from peer 2145d15ab2d74c0287839266f1411a43 ({username='slave'} at 127.0.99.130:49785)
I20240503 13:59:55.026017 9362 tablet_copy_service.cc:161] P 0d77752181e944698e809e325e5937e6: Beginning new tablet copy session on tablet 6ee85f4483004c1f9499203c7323706d from peer 2145d15ab2d74c0287839266f1411a43 at {username='slave'} at 127.0.99.130:49785: session id = 2145d15ab2d74c0287839266f1411a43-6ee85f4483004c1f9499203c7323706d
I20240503 13:59:55.026952 9362 tablet_copy_source_session.cc:205] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:55.027877 13635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ee85f4483004c1f9499203c7323706d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:55.030464 13635 tablet_copy_client.cc:783] T 6ee85f4483004c1f9499203c7323706d P 2145d15ab2d74c0287839266f1411a43: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:55.030679 13635 tablet_copy_client.cc:647] T 6ee85f4483004c1f9499203c7323706d P 2145d15ab2d74c0287839266f1411a43: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:55.031780 13635 tablet_copy_client.cc:523] T 6ee85f4483004c1f9499203c7323706d P 2145d15ab2d74c0287839266f1411a43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:55.032975 13635 tablet_bootstrap.cc:492] T 6ee85f4483004c1f9499203c7323706d P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:55.037678 13635 tablet_bootstrap.cc:492] T 6ee85f4483004c1f9499203c7323706d P 2145d15ab2d74c0287839266f1411a43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:55.037920 13635 tablet_bootstrap.cc:492] T 6ee85f4483004c1f9499203c7323706d P 2145d15ab2d74c0287839266f1411a43: Bootstrap complete.
I20240503 13:59:55.038057 13635 ts_tablet_manager.cc:1367] T 6ee85f4483004c1f9499203c7323706d P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 13:59:55.038472 13635 raft_consensus.cc:348] T 6ee85f4483004c1f9499203c7323706d P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:55.038587 13635 raft_consensus.cc:729] T 6ee85f4483004c1f9499203c7323706d P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: LEARNER
I20240503 13:59:55.038668 13635 consensus_queue.cc:260] T 6ee85f4483004c1f9499203c7323706d P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:55.043777 13635 ts_tablet_manager.cc:1398] T 6ee85f4483004c1f9499203c7323706d P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:59:55.047307 9362 tablet_copy_service.cc:342] P 0d77752181e944698e809e325e5937e6: Request end of tablet copy session 2145d15ab2d74c0287839266f1411a43-6ee85f4483004c1f9499203c7323706d received from {username='slave'} at 127.0.99.130:49785
I20240503 13:59:55.047379 9362 tablet_copy_service.cc:433] P 0d77752181e944698e809e325e5937e6: ending tablet copy session 2145d15ab2d74c0287839266f1411a43-6ee85f4483004c1f9499203c7323706d on tablet 6ee85f4483004c1f9499203c7323706d with peer 2145d15ab2d74c0287839266f1411a43
I20240503 13:59:55.103497 6922 consensus_queue.cc:237] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:55.104902 9081 raft_consensus.cc:1261] T d002f7a57be04915b7b0b3dcb985f550 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20240503 13:59:55.105228 6856 consensus_peers.cc:485] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d002f7a57be04915b7b0b3dcb985f550. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:55.105350 9205 raft_consensus.cc:1261] T d002f7a57be04915b7b0b3dcb985f550 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:55.105450 13370 consensus_queue.cc:1035] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:55.106827 13370 consensus_queue.cc:1035] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:55.108522 11311 raft_consensus.cc:2932] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index -1 to 3, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:55.109341 9205 raft_consensus.cc:2932] T d002f7a57be04915b7b0b3dcb985f550 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index -1 to 3, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:55.112226 6385 catalog_manager.cc:5484] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index -1 to 3, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:55.112267 9081 raft_consensus.cc:2932] T d002f7a57be04915b7b0b3dcb985f550 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index -1 to 3, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:55.115361 13540 ts_tablet_manager.cc:902] T aecb348d4b3b4500a587c92a7fefab4e P f58065db70b64ef8b1dc499aa6ae4460: Initiating tablet copy from peer 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:55.115851 13540 tablet_copy_client.cc:308] T aecb348d4b3b4500a587c92a7fefab4e P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38021
I20240503 13:59:55.116257 6549 tablet_copy_service.cc:140] P 5f46e19644fd4f4b825c86ba6af768da: Received BeginTabletCopySession request for tablet aecb348d4b3b4500a587c92a7fefab4e from peer f58065db70b64ef8b1dc499aa6ae4460 ({username='slave'} at 127.0.99.132:48317)
I20240503 13:59:55.116361 6549 tablet_copy_service.cc:161] P 5f46e19644fd4f4b825c86ba6af768da: Beginning new tablet copy session on tablet aecb348d4b3b4500a587c92a7fefab4e from peer f58065db70b64ef8b1dc499aa6ae4460 at {username='slave'} at 127.0.99.132:48317: session id = f58065db70b64ef8b1dc499aa6ae4460-aecb348d4b3b4500a587c92a7fefab4e
I20240503 13:59:55.117246 6549 tablet_copy_source_session.cc:205] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:55.119928 13540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aecb348d4b3b4500a587c92a7fefab4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:55.122063 13540 tablet_copy_client.cc:783] T aecb348d4b3b4500a587c92a7fefab4e P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:55.122211 13540 tablet_copy_client.cc:647] T aecb348d4b3b4500a587c92a7fefab4e P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:55.123569 13540 tablet_copy_client.cc:523] T aecb348d4b3b4500a587c92a7fefab4e P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:55.125038 13540 tablet_bootstrap.cc:492] T aecb348d4b3b4500a587c92a7fefab4e P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:55.130443 6922 consensus_queue.cc:237] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:55.132033 8947 raft_consensus.cc:1261] T 218b25613baf48319576e433c2ac4534 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:55.132076 6528 raft_consensus.cc:1261] T 218b25613baf48319576e433c2ac4534 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:55.132443 13365 consensus_queue.cc:1035] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:55.132648 12748 consensus_queue.cc:1035] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 13:59:55.133934 6857 consensus_peers.cc:485] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): Couldn't send request to peer ab1d21935d6049888eb0ceccc3c9d735. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 218b25613baf48319576e433c2ac4534. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:55.134442 13365 raft_consensus.cc:2932] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:55.134863 6528 raft_consensus.cc:2932] T 218b25613baf48319576e433c2ac4534 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:55.138538 6381 catalog_manager.cc:5484] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index -1 to 2, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:55.139238 6788 raft_consensus.cc:1261] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 2 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 13:59:55.140607 12372 consensus_queue.cc:1035] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:55.141429 13540 tablet_bootstrap.cc:492] T aecb348d4b3b4500a587c92a7fefab4e P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:55.142323 13540 tablet_bootstrap.cc:492] T aecb348d4b3b4500a587c92a7fefab4e P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap complete.
I20240503 13:59:55.142513 13540 ts_tablet_manager.cc:1367] T aecb348d4b3b4500a587c92a7fefab4e P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.003s
I20240503 13:59:55.143066 13540 raft_consensus.cc:348] T aecb348d4b3b4500a587c92a7fefab4e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:55.143234 13540 raft_consensus.cc:729] T aecb348d4b3b4500a587c92a7fefab4e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: LEARNER
I20240503 13:59:55.143410 13540 consensus_queue.cc:260] T aecb348d4b3b4500a587c92a7fefab4e P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:55.143829 13540 ts_tablet_manager.cc:1398] T aecb348d4b3b4500a587c92a7fefab4e P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:55.145187 6549 tablet_copy_service.cc:342] P 5f46e19644fd4f4b825c86ba6af768da: Request end of tablet copy session f58065db70b64ef8b1dc499aa6ae4460-aecb348d4b3b4500a587c92a7fefab4e received from {username='slave'} at 127.0.99.132:48317
I20240503 13:59:55.145270 6549 tablet_copy_service.cc:433] P 5f46e19644fd4f4b825c86ba6af768da: ending tablet copy session f58065db70b64ef8b1dc499aa6ae4460-aecb348d4b3b4500a587c92a7fefab4e on tablet aecb348d4b3b4500a587c92a7fefab4e with peer f58065db70b64ef8b1dc499aa6ae4460
I20240503 13:59:55.148764 8947 raft_consensus.cc:2932] T 218b25613baf48319576e433c2ac4534 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:55.153095 6922 raft_consensus.cc:1261] T 64135eb6ef574c95b3677e32d66748a4 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 13:59:55.153961 12372 consensus_queue.cc:1035] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:55.157647 9342 raft_consensus.cc:1261] T 64135eb6ef574c95b3677e32d66748a4 P 0d77752181e944698e809e325e5937e6 [term 2 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 13:59:55.159035 13280 consensus_queue.cc:1035] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:55.191160 13751 ts_tablet_manager.cc:902] T 400c54bdecfa44e08c7eca046bcd2ffc P 8d574fb4a89d4eea947c859470d44dd4: Initiating tablet copy from peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:55.191694 13751 tablet_copy_client.cc:308] T 400c54bdecfa44e08c7eca046bcd2ffc P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43531
I20240503 13:59:55.192185 6942 tablet_copy_service.cc:140] P f58065db70b64ef8b1dc499aa6ae4460: Received BeginTabletCopySession request for tablet 400c54bdecfa44e08c7eca046bcd2ffc from peer 8d574fb4a89d4eea947c859470d44dd4 ({username='slave'} at 127.0.99.134:57297)
I20240503 13:59:55.192276 6942 tablet_copy_service.cc:161] P f58065db70b64ef8b1dc499aa6ae4460: Beginning new tablet copy session on tablet 400c54bdecfa44e08c7eca046bcd2ffc from peer 8d574fb4a89d4eea947c859470d44dd4 at {username='slave'} at 127.0.99.134:57297: session id = 8d574fb4a89d4eea947c859470d44dd4-400c54bdecfa44e08c7eca046bcd2ffc
I20240503 13:59:55.193217 6942 tablet_copy_source_session.cc:205] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:55.194418 13751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 400c54bdecfa44e08c7eca046bcd2ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:55.196787 13751 tablet_copy_client.cc:783] T 400c54bdecfa44e08c7eca046bcd2ffc P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:55.197000 13751 tablet_copy_client.cc:647] T 400c54bdecfa44e08c7eca046bcd2ffc P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:55.198160 13751 tablet_copy_client.cc:523] T 400c54bdecfa44e08c7eca046bcd2ffc P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:55.199391 13751 tablet_bootstrap.cc:492] T 400c54bdecfa44e08c7eca046bcd2ffc P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:55.224227 13751 tablet_bootstrap.cc:492] T 400c54bdecfa44e08c7eca046bcd2ffc P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:55.224591 13751 tablet_bootstrap.cc:492] T 400c54bdecfa44e08c7eca046bcd2ffc P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap complete.
I20240503 13:59:55.224790 13751 ts_tablet_manager.cc:1367] T 400c54bdecfa44e08c7eca046bcd2ffc P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.025s user 0.023s sys 0.000s
I20240503 13:59:55.225248 13751 raft_consensus.cc:348] T 400c54bdecfa44e08c7eca046bcd2ffc P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:55.225387 13751 raft_consensus.cc:729] T 400c54bdecfa44e08c7eca046bcd2ffc P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: LEARNER
I20240503 13:59:55.225507 13751 consensus_queue.cc:260] T 400c54bdecfa44e08c7eca046bcd2ffc P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:55.225848 13751 ts_tablet_manager.cc:1398] T 400c54bdecfa44e08c7eca046bcd2ffc P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:55.226704 6942 tablet_copy_service.cc:342] P f58065db70b64ef8b1dc499aa6ae4460: Request end of tablet copy session 8d574fb4a89d4eea947c859470d44dd4-400c54bdecfa44e08c7eca046bcd2ffc received from {username='slave'} at 127.0.99.134:57297
I20240503 13:59:55.226792 6942 tablet_copy_service.cc:433] P f58065db70b64ef8b1dc499aa6ae4460: ending tablet copy session 8d574fb4a89d4eea947c859470d44dd4-400c54bdecfa44e08c7eca046bcd2ffc on tablet 400c54bdecfa44e08c7eca046bcd2ffc with peer 8d574fb4a89d4eea947c859470d44dd4
I20240503 13:59:55.273669 13822 ts_tablet_manager.cc:902] T 44eddd53c3f348a9a1117a20473cbb65 P 5f46e19644fd4f4b825c86ba6af768da: Initiating tablet copy from peer 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:55.274422 13822 tablet_copy_client.cc:308] T 44eddd53c3f348a9a1117a20473cbb65 P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:39481
I20240503 13:59:55.274967 9101 tablet_copy_service.cc:140] P 8d574fb4a89d4eea947c859470d44dd4: Received BeginTabletCopySession request for tablet 44eddd53c3f348a9a1117a20473cbb65 from peer 5f46e19644fd4f4b825c86ba6af768da ({username='slave'} at 127.0.99.129:55581)
I20240503 13:59:55.275079 9101 tablet_copy_service.cc:161] P 8d574fb4a89d4eea947c859470d44dd4: Beginning new tablet copy session on tablet 44eddd53c3f348a9a1117a20473cbb65 from peer 5f46e19644fd4f4b825c86ba6af768da at {username='slave'} at 127.0.99.129:55581: session id = 5f46e19644fd4f4b825c86ba6af768da-44eddd53c3f348a9a1117a20473cbb65
I20240503 13:59:55.276130 9101 tablet_copy_source_session.cc:205] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:55.276974 13822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44eddd53c3f348a9a1117a20473cbb65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:55.279181 13822 tablet_copy_client.cc:783] T 44eddd53c3f348a9a1117a20473cbb65 P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:55.279349 13822 tablet_copy_client.cc:647] T 44eddd53c3f348a9a1117a20473cbb65 P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:55.282930 13822 tablet_copy_client.cc:523] T 44eddd53c3f348a9a1117a20473cbb65 P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:55.284188 13822 tablet_bootstrap.cc:492] T 44eddd53c3f348a9a1117a20473cbb65 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:55.291139 13751 ts_tablet_manager.cc:902] T 01551845458048e48df9901947b4eec6 P 8d574fb4a89d4eea947c859470d44dd4: Initiating tablet copy from peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:55.291744 13751 tablet_copy_client.cc:308] T 01551845458048e48df9901947b4eec6 P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:46365
I20240503 13:59:55.292006 13822 tablet_bootstrap.cc:492] T 44eddd53c3f348a9a1117a20473cbb65 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:55.292253 13822 tablet_bootstrap.cc:492] T 44eddd53c3f348a9a1117a20473cbb65 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap complete.
I20240503 13:59:55.292363 6811 tablet_copy_service.cc:140] P 4a85e492c8154e9881a9cc2a105d7e64: Received BeginTabletCopySession request for tablet 01551845458048e48df9901947b4eec6 from peer 8d574fb4a89d4eea947c859470d44dd4 ({username='slave'} at 127.0.99.134:52175)
I20240503 13:59:55.292419 13822 ts_tablet_manager.cc:1367] T 44eddd53c3f348a9a1117a20473cbb65 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20240503 13:59:55.292487 6811 tablet_copy_service.cc:161] P 4a85e492c8154e9881a9cc2a105d7e64: Beginning new tablet copy session on tablet 01551845458048e48df9901947b4eec6 from peer 8d574fb4a89d4eea947c859470d44dd4 at {username='slave'} at 127.0.99.134:52175: session id = 8d574fb4a89d4eea947c859470d44dd4-01551845458048e48df9901947b4eec6
I20240503 13:59:55.292882 13822 raft_consensus.cc:348] T 44eddd53c3f348a9a1117a20473cbb65 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:55.292970 13822 raft_consensus.cc:729] T 44eddd53c3f348a9a1117a20473cbb65 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: LEARNER
I20240503 13:59:55.293053 13822 consensus_queue.cc:260] T 44eddd53c3f348a9a1117a20473cbb65 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:55.293370 6811 tablet_copy_source_session.cc:205] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:55.293505 13822 ts_tablet_manager.cc:1398] T 44eddd53c3f348a9a1117a20473cbb65 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:55.294230 13751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01551845458048e48df9901947b4eec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:55.294484 9101 tablet_copy_service.cc:342] P 8d574fb4a89d4eea947c859470d44dd4: Request end of tablet copy session 5f46e19644fd4f4b825c86ba6af768da-44eddd53c3f348a9a1117a20473cbb65 received from {username='slave'} at 127.0.99.129:55581
I20240503 13:59:55.294589 9101 tablet_copy_service.cc:433] P 8d574fb4a89d4eea947c859470d44dd4: ending tablet copy session 5f46e19644fd4f4b825c86ba6af768da-44eddd53c3f348a9a1117a20473cbb65 on tablet 44eddd53c3f348a9a1117a20473cbb65 with peer 5f46e19644fd4f4b825c86ba6af768da
I20240503 13:59:55.296466 13751 tablet_copy_client.cc:783] T 01551845458048e48df9901947b4eec6 P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:55.296622 13751 tablet_copy_client.cc:647] T 01551845458048e48df9901947b4eec6 P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:55.298132 13751 tablet_copy_client.cc:523] T 01551845458048e48df9901947b4eec6 P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:55.299244 13751 tablet_bootstrap.cc:492] T 01551845458048e48df9901947b4eec6 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:55.302417 13751 tablet_bootstrap.cc:492] T 01551845458048e48df9901947b4eec6 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:55.302712 13751 tablet_bootstrap.cc:492] T 01551845458048e48df9901947b4eec6 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap complete.
I20240503 13:59:55.302879 13751 ts_tablet_manager.cc:1367] T 01551845458048e48df9901947b4eec6 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 13:59:55.303346 13751 raft_consensus.cc:348] T 01551845458048e48df9901947b4eec6 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:55.303478 13751 raft_consensus.cc:729] T 01551845458048e48df9901947b4eec6 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: LEARNER
I20240503 13:59:55.303593 13751 consensus_queue.cc:260] T 01551845458048e48df9901947b4eec6 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:55.304672 13751 ts_tablet_manager.cc:1398] T 01551845458048e48df9901947b4eec6 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:55.305069 6811 tablet_copy_service.cc:342] P 4a85e492c8154e9881a9cc2a105d7e64: Request end of tablet copy session 8d574fb4a89d4eea947c859470d44dd4-01551845458048e48df9901947b4eec6 received from {username='slave'} at 127.0.99.134:52175
I20240503 13:59:55.305150 6811 tablet_copy_service.cc:433] P 4a85e492c8154e9881a9cc2a105d7e64: ending tablet copy session 8d574fb4a89d4eea947c859470d44dd4-01551845458048e48df9901947b4eec6 on tablet 01551845458048e48df9901947b4eec6 with peer 8d574fb4a89d4eea947c859470d44dd4
I20240503 13:59:55.332274 6922 raft_consensus.cc:1203] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:55.338480 6922 raft_consensus.cc:1203] T d588a730a0c94d4f9446049e537c6727 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:55.348601 13015 raft_consensus.cc:1050] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545: attempting to promote NON_VOTER 2145d15ab2d74c0287839266f1411a43 to VOTER
I20240503 13:59:55.349042 13015 consensus_queue.cc:237] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 13:59:55.350517 9081 raft_consensus.cc:1261] T a16acc33ab6c45858c52cceb4355207a P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:55.350891 13268 consensus_queue.cc:1035] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:55.351358 9342 raft_consensus.cc:1261] T a16acc33ab6c45858c52cceb4355207a P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:55.351830 13268 consensus_queue.cc:1035] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:55.352669 6657 raft_consensus.cc:1261] T a16acc33ab6c45858c52cceb4355207a P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:55.353205 13268 consensus_queue.cc:1035] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:55.354280 13268 raft_consensus.cc:2932] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:55.354882 9342 raft_consensus.cc:2932] T a16acc33ab6c45858c52cceb4355207a P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:55.356312 9079 raft_consensus.cc:2932] T a16acc33ab6c45858c52cceb4355207a P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:55.357930 6387 catalog_manager.cc:5484] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:55.359467 6657 raft_consensus.cc:2932] T a16acc33ab6c45858c52cceb4355207a P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:55.364626 8947 consensus_queue.cc:237] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 13:59:55.365664 9079 raft_consensus.cc:1261] T a16acc33ab6c45858c52cceb4355207a P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:55.365655 6657 raft_consensus.cc:1261] T a16acc33ab6c45858c52cceb4355207a P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:55.366037 11750 consensus_queue.cc:1035] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:55.366230 13015 consensus_queue.cc:1035] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:55.367704 13015 raft_consensus.cc:2932] T a16acc33ab6c45858c52cceb4355207a P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:55.368038 6657 raft_consensus.cc:2932] T a16acc33ab6c45858c52cceb4355207a P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:55.368232 9079 raft_consensus.cc:2932] T a16acc33ab6c45858c52cceb4355207a P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:55.369437 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a16acc33ab6c45858c52cceb4355207a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:55.370146 13842 ts_tablet_manager.cc:902] T 4a1398d983c54a7498257f766052e8e9 P 4a85e492c8154e9881a9cc2a105d7e64: Initiating tablet copy from peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:55.370657 13842 tablet_copy_client.cc:308] T 4a1398d983c54a7498257f766052e8e9 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:41931
I20240503 13:59:55.371263 9362 tablet_copy_service.cc:140] P 0d77752181e944698e809e325e5937e6: Received BeginTabletCopySession request for tablet 4a1398d983c54a7498257f766052e8e9 from peer 4a85e492c8154e9881a9cc2a105d7e64 ({username='slave'} at 127.0.99.131:49089)
I20240503 13:59:55.371378 9362 tablet_copy_service.cc:161] P 0d77752181e944698e809e325e5937e6: Beginning new tablet copy session on tablet 4a1398d983c54a7498257f766052e8e9 from peer 4a85e492c8154e9881a9cc2a105d7e64 at {username='slave'} at 127.0.99.131:49089: session id = 4a85e492c8154e9881a9cc2a105d7e64-4a1398d983c54a7498257f766052e8e9
I20240503 13:59:55.371932 6381 catalog_manager.cc:5484] T a16acc33ab6c45858c52cceb4355207a P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:55.372229 9362 tablet_copy_source_session.cc:205] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:55.373834 13842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a1398d983c54a7498257f766052e8e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:55.375798 9321 tablet_service.cc:1514] Processing DeleteTablet for tablet a16acc33ab6c45858c52cceb4355207a with delete_type TABLET_DATA_TOMBSTONED (TS 0d77752181e944698e809e325e5937e6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:55.376453 13847 tablet_replica.cc:330] T a16acc33ab6c45858c52cceb4355207a P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:55.376623 13847 raft_consensus.cc:2229] T a16acc33ab6c45858c52cceb4355207a P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:55.376782 13847 raft_consensus.cc:2258] T a16acc33ab6c45858c52cceb4355207a P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.377095 13842 tablet_copy_client.cc:783] T 4a1398d983c54a7498257f766052e8e9 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:55.377250 13842 tablet_copy_client.cc:647] T 4a1398d983c54a7498257f766052e8e9 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:55.378355 13847 ts_tablet_manager.cc:1875] T a16acc33ab6c45858c52cceb4355207a P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:55.379392 13842 tablet_copy_client.cc:523] T 4a1398d983c54a7498257f766052e8e9 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:55.380690 13842 tablet_bootstrap.cc:492] T 4a1398d983c54a7498257f766052e8e9 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:55.380805 13847 ts_tablet_manager.cc:1888] T a16acc33ab6c45858c52cceb4355207a P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:55.380880 13847 log.cc:1196] T a16acc33ab6c45858c52cceb4355207a P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/a16acc33ab6c45858c52cceb4355207a
I20240503 13:59:55.381008 13848 ts_tablet_manager.cc:902] T ef19b1e5f15e49448c1b2919a24090d3 P 0d77752181e944698e809e325e5937e6: Initiating tablet copy from peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:55.381426 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet a16acc33ab6c45858c52cceb4355207a (table rebalancer_extra_table_24 [id=595e2f47ca074ec4bd181e870ba60ff9]) successfully deleted
I20240503 13:59:55.381505 13848 tablet_copy_client.cc:308] T ef19b1e5f15e49448c1b2919a24090d3 P 0d77752181e944698e809e325e5937e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:34547
I20240503 13:59:55.381909 8968 tablet_copy_service.cc:140] P 06ca8dbc004c42b99faa527a4e6f9545: Received BeginTabletCopySession request for tablet ef19b1e5f15e49448c1b2919a24090d3 from peer 0d77752181e944698e809e325e5937e6 ({username='slave'} at 127.0.99.136:60577)
I20240503 13:59:55.381989 8968 tablet_copy_service.cc:161] P 06ca8dbc004c42b99faa527a4e6f9545: Beginning new tablet copy session on tablet ef19b1e5f15e49448c1b2919a24090d3 from peer 0d77752181e944698e809e325e5937e6 at {username='slave'} at 127.0.99.136:60577: session id = 0d77752181e944698e809e325e5937e6-ef19b1e5f15e49448c1b2919a24090d3
I20240503 13:59:55.382710 8968 tablet_copy_source_session.cc:205] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:55.383487 13848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef19b1e5f15e49448c1b2919a24090d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:55.384399 13842 tablet_bootstrap.cc:492] T 4a1398d983c54a7498257f766052e8e9 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:55.384671 13842 tablet_bootstrap.cc:492] T 4a1398d983c54a7498257f766052e8e9 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap complete.
I20240503 13:59:55.384845 13842 ts_tablet_manager.cc:1367] T 4a1398d983c54a7498257f766052e8e9 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20240503 13:59:55.385347 13842 raft_consensus.cc:348] T 4a1398d983c54a7498257f766052e8e9 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:55.385481 13842 raft_consensus.cc:729] T 4a1398d983c54a7498257f766052e8e9 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: LEARNER
I20240503 13:59:55.385632 13848 tablet_copy_client.cc:783] T ef19b1e5f15e49448c1b2919a24090d3 P 0d77752181e944698e809e325e5937e6: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:55.385588 13842 consensus_queue.cc:260] T 4a1398d983c54a7498257f766052e8e9 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:55.385785 13848 tablet_copy_client.cc:647] T ef19b1e5f15e49448c1b2919a24090d3 P 0d77752181e944698e809e325e5937e6: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:55.386011 13842 ts_tablet_manager.cc:1398] T 4a1398d983c54a7498257f766052e8e9 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:55.386564 9362 tablet_copy_service.cc:342] P 0d77752181e944698e809e325e5937e6: Request end of tablet copy session 4a85e492c8154e9881a9cc2a105d7e64-4a1398d983c54a7498257f766052e8e9 received from {username='slave'} at 127.0.99.131:49089
I20240503 13:59:55.386629 9362 tablet_copy_service.cc:433] P 0d77752181e944698e809e325e5937e6: ending tablet copy session 4a85e492c8154e9881a9cc2a105d7e64-4a1398d983c54a7498257f766052e8e9 on tablet 4a1398d983c54a7498257f766052e8e9 with peer 4a85e492c8154e9881a9cc2a105d7e64
I20240503 13:59:55.388422 13848 tablet_copy_client.cc:523] T ef19b1e5f15e49448c1b2919a24090d3 P 0d77752181e944698e809e325e5937e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:55.389467 13848 tablet_bootstrap.cc:492] T ef19b1e5f15e49448c1b2919a24090d3 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:55.392585 13848 tablet_bootstrap.cc:492] T ef19b1e5f15e49448c1b2919a24090d3 P 0d77752181e944698e809e325e5937e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:55.392817 13848 tablet_bootstrap.cc:492] T ef19b1e5f15e49448c1b2919a24090d3 P 0d77752181e944698e809e325e5937e6: Bootstrap complete.
I20240503 13:59:55.392974 13848 ts_tablet_manager.cc:1367] T ef19b1e5f15e49448c1b2919a24090d3 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 13:59:55.393366 13848 raft_consensus.cc:348] T ef19b1e5f15e49448c1b2919a24090d3 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:55.393486 13848 raft_consensus.cc:729] T ef19b1e5f15e49448c1b2919a24090d3 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: LEARNER
I20240503 13:59:55.393566 13848 consensus_queue.cc:260] T ef19b1e5f15e49448c1b2919a24090d3 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:55.393801 13848 ts_tablet_manager.cc:1398] T ef19b1e5f15e49448c1b2919a24090d3 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:55.394287 8968 tablet_copy_service.cc:342] P 06ca8dbc004c42b99faa527a4e6f9545: Request end of tablet copy session 0d77752181e944698e809e325e5937e6-ef19b1e5f15e49448c1b2919a24090d3 received from {username='slave'} at 127.0.99.136:60577
I20240503 13:59:55.394352 8968 tablet_copy_service.cc:433] P 06ca8dbc004c42b99faa527a4e6f9545: ending tablet copy session 0d77752181e944698e809e325e5937e6-ef19b1e5f15e49448c1b2919a24090d3 on tablet ef19b1e5f15e49448c1b2919a24090d3 with peer 0d77752181e944698e809e325e5937e6
I20240503 13:59:55.397365 13855 ts_tablet_manager.cc:902] T da69140936424a31b66c87e2ad4092cb P ab1d21935d6049888eb0ceccc3c9d735: Initiating tablet copy from peer 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:55.397843 13855 tablet_copy_client.cc:308] T da69140936424a31b66c87e2ad4092cb P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38021
I20240503 13:59:55.398280 6549 tablet_copy_service.cc:140] P 5f46e19644fd4f4b825c86ba6af768da: Received BeginTabletCopySession request for tablet da69140936424a31b66c87e2ad4092cb from peer ab1d21935d6049888eb0ceccc3c9d735 ({username='slave'} at 127.0.99.135:46495)
I20240503 13:59:55.398378 6549 tablet_copy_service.cc:161] P 5f46e19644fd4f4b825c86ba6af768da: Beginning new tablet copy session on tablet da69140936424a31b66c87e2ad4092cb from peer ab1d21935d6049888eb0ceccc3c9d735 at {username='slave'} at 127.0.99.135:46495: session id = ab1d21935d6049888eb0ceccc3c9d735-da69140936424a31b66c87e2ad4092cb
I20240503 13:59:55.399194 6549 tablet_copy_source_session.cc:205] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:55.399969 13855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da69140936424a31b66c87e2ad4092cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:55.401690 13855 tablet_copy_client.cc:783] T da69140936424a31b66c87e2ad4092cb P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:55.401825 13855 tablet_copy_client.cc:647] T da69140936424a31b66c87e2ad4092cb P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:55.402899 13855 tablet_copy_client.cc:523] T da69140936424a31b66c87e2ad4092cb P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:55.404198 13855 tablet_bootstrap.cc:492] T da69140936424a31b66c87e2ad4092cb P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:55.406996 13848 ts_tablet_manager.cc:902] T e106b1432baf423493c95fcce63a55da P 0d77752181e944698e809e325e5937e6: Initiating tablet copy from peer 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:55.407430 13848 tablet_copy_client.cc:308] T e106b1432baf423493c95fcce63a55da P 0d77752181e944698e809e325e5937e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38021
I20240503 13:59:55.407832 6549 tablet_copy_service.cc:140] P 5f46e19644fd4f4b825c86ba6af768da: Received BeginTabletCopySession request for tablet e106b1432baf423493c95fcce63a55da from peer 0d77752181e944698e809e325e5937e6 ({username='slave'} at 127.0.99.136:52973)
I20240503 13:59:55.407927 6549 tablet_copy_service.cc:161] P 5f46e19644fd4f4b825c86ba6af768da: Beginning new tablet copy session on tablet e106b1432baf423493c95fcce63a55da from peer 0d77752181e944698e809e325e5937e6 at {username='slave'} at 127.0.99.136:52973: session id = 0d77752181e944698e809e325e5937e6-e106b1432baf423493c95fcce63a55da
I20240503 13:59:55.408954 6549 tablet_copy_source_session.cc:205] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:55.409750 13848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e106b1432baf423493c95fcce63a55da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:55.411914 13848 tablet_copy_client.cc:783] T e106b1432baf423493c95fcce63a55da P 0d77752181e944698e809e325e5937e6: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:55.412060 13848 tablet_copy_client.cc:647] T e106b1432baf423493c95fcce63a55da P 0d77752181e944698e809e325e5937e6: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:55.413271 13848 tablet_copy_client.cc:523] T e106b1432baf423493c95fcce63a55da P 0d77752181e944698e809e325e5937e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:55.414429 13848 tablet_bootstrap.cc:492] T e106b1432baf423493c95fcce63a55da P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:55.430521 13855 tablet_bootstrap.cc:492] T da69140936424a31b66c87e2ad4092cb P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:55.430822 13855 tablet_bootstrap.cc:492] T da69140936424a31b66c87e2ad4092cb P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap complete.
I20240503 13:59:55.431025 13855 ts_tablet_manager.cc:1367] T da69140936424a31b66c87e2ad4092cb P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.027s user 0.022s sys 0.000s
I20240503 13:59:55.431481 13855 raft_consensus.cc:348] T da69140936424a31b66c87e2ad4092cb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:55.431607 13855 raft_consensus.cc:729] T da69140936424a31b66c87e2ad4092cb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: LEARNER
I20240503 13:59:55.431696 13855 consensus_queue.cc:260] T da69140936424a31b66c87e2ad4092cb P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:55.432150 13855 ts_tablet_manager.cc:1398] T da69140936424a31b66c87e2ad4092cb P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240503 13:59:55.432603 6549 tablet_copy_service.cc:342] P 5f46e19644fd4f4b825c86ba6af768da: Request end of tablet copy session ab1d21935d6049888eb0ceccc3c9d735-da69140936424a31b66c87e2ad4092cb received from {username='slave'} at 127.0.99.135:46495
I20240503 13:59:55.432686 6549 tablet_copy_service.cc:433] P 5f46e19644fd4f4b825c86ba6af768da: ending tablet copy session ab1d21935d6049888eb0ceccc3c9d735-da69140936424a31b66c87e2ad4092cb on tablet da69140936424a31b66c87e2ad4092cb with peer ab1d21935d6049888eb0ceccc3c9d735
I20240503 13:59:55.443207 13540 ts_tablet_manager.cc:902] T 61dfb1d40ee444368470a0dafc3179a4 P f58065db70b64ef8b1dc499aa6ae4460: Initiating tablet copy from peer 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:55.443688 13540 tablet_copy_client.cc:308] T 61dfb1d40ee444368470a0dafc3179a4 P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46119
I20240503 13:59:55.444145 6680 tablet_copy_service.cc:140] P 2145d15ab2d74c0287839266f1411a43: Received BeginTabletCopySession request for tablet 61dfb1d40ee444368470a0dafc3179a4 from peer f58065db70b64ef8b1dc499aa6ae4460 ({username='slave'} at 127.0.99.132:45437)
I20240503 13:59:55.444240 6680 tablet_copy_service.cc:161] P 2145d15ab2d74c0287839266f1411a43: Beginning new tablet copy session on tablet 61dfb1d40ee444368470a0dafc3179a4 from peer f58065db70b64ef8b1dc499aa6ae4460 at {username='slave'} at 127.0.99.132:45437: session id = f58065db70b64ef8b1dc499aa6ae4460-61dfb1d40ee444368470a0dafc3179a4
I20240503 13:59:55.444777 13848 tablet_bootstrap.cc:492] T e106b1432baf423493c95fcce63a55da P 0d77752181e944698e809e325e5937e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:55.444983 6680 tablet_copy_source_session.cc:205] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:55.445047 13848 tablet_bootstrap.cc:492] T e106b1432baf423493c95fcce63a55da P 0d77752181e944698e809e325e5937e6: Bootstrap complete.
I20240503 13:59:55.445206 13848 ts_tablet_manager.cc:1367] T e106b1432baf423493c95fcce63a55da P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.031s user 0.026s sys 0.001s
I20240503 13:59:55.445652 13848 raft_consensus.cc:348] T e106b1432baf423493c95fcce63a55da P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:55.445755 13848 raft_consensus.cc:729] T e106b1432baf423493c95fcce63a55da P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: LEARNER
I20240503 13:59:55.445824 13540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61dfb1d40ee444368470a0dafc3179a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:55.445843 13848 consensus_queue.cc:260] T e106b1432baf423493c95fcce63a55da P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:55.446116 13848 ts_tablet_manager.cc:1398] T e106b1432baf423493c95fcce63a55da P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:55.446578 6549 tablet_copy_service.cc:342] P 5f46e19644fd4f4b825c86ba6af768da: Request end of tablet copy session 0d77752181e944698e809e325e5937e6-e106b1432baf423493c95fcce63a55da received from {username='slave'} at 127.0.99.136:52973
I20240503 13:59:55.446648 6549 tablet_copy_service.cc:433] P 5f46e19644fd4f4b825c86ba6af768da: ending tablet copy session 0d77752181e944698e809e325e5937e6-e106b1432baf423493c95fcce63a55da on tablet e106b1432baf423493c95fcce63a55da with peer 0d77752181e944698e809e325e5937e6
I20240503 13:59:55.448120 13540 tablet_copy_client.cc:783] T 61dfb1d40ee444368470a0dafc3179a4 P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:55.448272 13540 tablet_copy_client.cc:647] T 61dfb1d40ee444368470a0dafc3179a4 P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:55.449530 13540 tablet_copy_client.cc:523] T 61dfb1d40ee444368470a0dafc3179a4 P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:55.450798 13540 tablet_bootstrap.cc:492] T 61dfb1d40ee444368470a0dafc3179a4 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:55.453703 13540 tablet_bootstrap.cc:492] T 61dfb1d40ee444368470a0dafc3179a4 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:55.453915 13540 tablet_bootstrap.cc:492] T 61dfb1d40ee444368470a0dafc3179a4 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap complete.
I20240503 13:59:55.454053 13540 ts_tablet_manager.cc:1367] T 61dfb1d40ee444368470a0dafc3179a4 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:59:55.454411 13540 raft_consensus.cc:348] T 61dfb1d40ee444368470a0dafc3179a4 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:55.454506 13540 raft_consensus.cc:729] T 61dfb1d40ee444368470a0dafc3179a4 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: LEARNER
I20240503 13:59:55.454607 13540 consensus_queue.cc:260] T 61dfb1d40ee444368470a0dafc3179a4 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:55.455164 13842 ts_tablet_manager.cc:902] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 4a85e492c8154e9881a9cc2a105d7e64: Initiating tablet copy from peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:55.455626 13842 tablet_copy_client.cc:308] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:41931
I20240503 13:59:55.456036 9362 tablet_copy_service.cc:140] P 0d77752181e944698e809e325e5937e6: Received BeginTabletCopySession request for tablet 7a25af1e7ae64f4fad8ad82faa1c6eb6 from peer 4a85e492c8154e9881a9cc2a105d7e64 ({username='slave'} at 127.0.99.131:49089)
I20240503 13:59:55.456146 9362 tablet_copy_service.cc:161] P 0d77752181e944698e809e325e5937e6: Beginning new tablet copy session on tablet 7a25af1e7ae64f4fad8ad82faa1c6eb6 from peer 4a85e492c8154e9881a9cc2a105d7e64 at {username='slave'} at 127.0.99.131:49089: session id = 4a85e492c8154e9881a9cc2a105d7e64-7a25af1e7ae64f4fad8ad82faa1c6eb6
I20240503 13:59:55.456789 9362 tablet_copy_source_session.cc:205] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:55.457270 13540 ts_tablet_manager.cc:1398] T 61dfb1d40ee444368470a0dafc3179a4 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:59:55.457729 6680 tablet_copy_service.cc:342] P 2145d15ab2d74c0287839266f1411a43: Request end of tablet copy session f58065db70b64ef8b1dc499aa6ae4460-61dfb1d40ee444368470a0dafc3179a4 received from {username='slave'} at 127.0.99.132:45437
I20240503 13:59:55.457803 13842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a25af1e7ae64f4fad8ad82faa1c6eb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:55.457818 6680 tablet_copy_service.cc:433] P 2145d15ab2d74c0287839266f1411a43: ending tablet copy session f58065db70b64ef8b1dc499aa6ae4460-61dfb1d40ee444368470a0dafc3179a4 on tablet 61dfb1d40ee444368470a0dafc3179a4 with peer f58065db70b64ef8b1dc499aa6ae4460
I20240503 13:59:55.459959 13842 tablet_copy_client.cc:783] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:55.460093 13842 tablet_copy_client.cc:647] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:55.461462 13842 tablet_copy_client.cc:523] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:55.462671 13842 tablet_bootstrap.cc:492] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:55.465953 13848 ts_tablet_manager.cc:902] T ea45c42d7fed4171a1b726b0684d2fa5 P 0d77752181e944698e809e325e5937e6: Initiating tablet copy from peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:55.466326 13842 tablet_bootstrap.cc:492] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:55.466358 13848 tablet_copy_client.cc:308] T ea45c42d7fed4171a1b726b0684d2fa5 P 0d77752181e944698e809e325e5937e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:46365
I20240503 13:59:55.466593 13842 tablet_bootstrap.cc:492] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap complete.
I20240503 13:59:55.466732 6811 tablet_copy_service.cc:140] P 4a85e492c8154e9881a9cc2a105d7e64: Received BeginTabletCopySession request for tablet ea45c42d7fed4171a1b726b0684d2fa5 from peer 0d77752181e944698e809e325e5937e6 ({username='slave'} at 127.0.99.136:34769)
I20240503 13:59:55.466779 13842 ts_tablet_manager.cc:1367] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 13:59:55.466820 6811 tablet_copy_service.cc:161] P 4a85e492c8154e9881a9cc2a105d7e64: Beginning new tablet copy session on tablet ea45c42d7fed4171a1b726b0684d2fa5 from peer 0d77752181e944698e809e325e5937e6 at {username='slave'} at 127.0.99.136:34769: session id = 0d77752181e944698e809e325e5937e6-ea45c42d7fed4171a1b726b0684d2fa5
I20240503 13:59:55.467167 13842 raft_consensus.cc:348] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:55.467269 13842 raft_consensus.cc:729] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: LEARNER
I20240503 13:59:55.467438 13842 consensus_queue.cc:260] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:55.467689 6811 tablet_copy_source_session.cc:205] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:55.467746 13842 ts_tablet_manager.cc:1398] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:55.468374 13848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea45c42d7fed4171a1b726b0684d2fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:55.468439 9362 tablet_copy_service.cc:342] P 0d77752181e944698e809e325e5937e6: Request end of tablet copy session 4a85e492c8154e9881a9cc2a105d7e64-7a25af1e7ae64f4fad8ad82faa1c6eb6 received from {username='slave'} at 127.0.99.131:49089
I20240503 13:59:55.468503 9362 tablet_copy_service.cc:433] P 0d77752181e944698e809e325e5937e6: ending tablet copy session 4a85e492c8154e9881a9cc2a105d7e64-7a25af1e7ae64f4fad8ad82faa1c6eb6 on tablet 7a25af1e7ae64f4fad8ad82faa1c6eb6 with peer 4a85e492c8154e9881a9cc2a105d7e64
I20240503 13:59:55.470793 13848 tablet_copy_client.cc:783] T ea45c42d7fed4171a1b726b0684d2fa5 P 0d77752181e944698e809e325e5937e6: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:55.470940 13848 tablet_copy_client.cc:647] T ea45c42d7fed4171a1b726b0684d2fa5 P 0d77752181e944698e809e325e5937e6: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:55.472182 13848 tablet_copy_client.cc:523] T ea45c42d7fed4171a1b726b0684d2fa5 P 0d77752181e944698e809e325e5937e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:55.473220 13848 tablet_bootstrap.cc:492] T ea45c42d7fed4171a1b726b0684d2fa5 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:55.481353 8947 raft_consensus.cc:1203] T f1291c3100bb4533891a1a7a3357d363 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:55.488808 13842 ts_tablet_manager.cc:902] T d002f7a57be04915b7b0b3dcb985f550 P 4a85e492c8154e9881a9cc2a105d7e64: Initiating tablet copy from peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:55.489297 13842 tablet_copy_client.cc:308] T d002f7a57be04915b7b0b3dcb985f550 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43531
I20240503 13:59:55.489717 6942 tablet_copy_service.cc:140] P f58065db70b64ef8b1dc499aa6ae4460: Received BeginTabletCopySession request for tablet d002f7a57be04915b7b0b3dcb985f550 from peer 4a85e492c8154e9881a9cc2a105d7e64 ({username='slave'} at 127.0.99.131:38291)
I20240503 13:59:55.489823 6942 tablet_copy_service.cc:161] P f58065db70b64ef8b1dc499aa6ae4460: Beginning new tablet copy session on tablet d002f7a57be04915b7b0b3dcb985f550 from peer 4a85e492c8154e9881a9cc2a105d7e64 at {username='slave'} at 127.0.99.131:38291: session id = 4a85e492c8154e9881a9cc2a105d7e64-d002f7a57be04915b7b0b3dcb985f550
I20240503 13:59:55.490782 6942 tablet_copy_source_session.cc:205] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:55.492405 13842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d002f7a57be04915b7b0b3dcb985f550. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:55.494534 13842 tablet_copy_client.cc:783] T d002f7a57be04915b7b0b3dcb985f550 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:55.494706 13842 tablet_copy_client.cc:647] T d002f7a57be04915b7b0b3dcb985f550 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:55.496806 13842 tablet_copy_client.cc:523] T d002f7a57be04915b7b0b3dcb985f550 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:55.497678 13848 tablet_bootstrap.cc:492] T ea45c42d7fed4171a1b726b0684d2fa5 P 0d77752181e944698e809e325e5937e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:55.497990 13842 tablet_bootstrap.cc:492] T d002f7a57be04915b7b0b3dcb985f550 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:55.498003 13848 tablet_bootstrap.cc:492] T ea45c42d7fed4171a1b726b0684d2fa5 P 0d77752181e944698e809e325e5937e6: Bootstrap complete.
I20240503 13:59:55.498155 13848 ts_tablet_manager.cc:1367] T ea45c42d7fed4171a1b726b0684d2fa5 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.025s user 0.019s sys 0.004s
I20240503 13:59:55.498631 13848 raft_consensus.cc:348] T ea45c42d7fed4171a1b726b0684d2fa5 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:55.498764 13848 raft_consensus.cc:729] T ea45c42d7fed4171a1b726b0684d2fa5 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: LEARNER
I20240503 13:59:55.498862 13848 consensus_queue.cc:260] T ea45c42d7fed4171a1b726b0684d2fa5 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:55.499173 13848 ts_tablet_manager.cc:1398] T ea45c42d7fed4171a1b726b0684d2fa5 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:55.499666 6811 tablet_copy_service.cc:342] P 4a85e492c8154e9881a9cc2a105d7e64: Request end of tablet copy session 0d77752181e944698e809e325e5937e6-ea45c42d7fed4171a1b726b0684d2fa5 received from {username='slave'} at 127.0.99.136:34769
I20240503 13:59:55.499737 6811 tablet_copy_service.cc:433] P 4a85e492c8154e9881a9cc2a105d7e64: ending tablet copy session 0d77752181e944698e809e325e5937e6-ea45c42d7fed4171a1b726b0684d2fa5 on tablet ea45c42d7fed4171a1b726b0684d2fa5 with peer 0d77752181e944698e809e325e5937e6
I20240503 13:59:55.506618 13842 tablet.cc:1722] T d002f7a57be04915b7b0b3dcb985f550 P 4a85e492c8154e9881a9cc2a105d7e64: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:59:55.506927 13842 tablet.cc:1620] T d002f7a57be04915b7b0b3dcb985f550 P 4a85e492c8154e9881a9cc2a105d7e64: MemRowSet was empty: no flush needed.
I20240503 13:59:55.507000 13842 tablet_metadata.cc:623] Not flushing: waiting for 1 pins to be released.
I20240503 13:59:55.507642 13842 tablet_bootstrap.cc:492] T d002f7a57be04915b7b0b3dcb985f550 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:55.507900 13842 tablet_bootstrap.cc:492] T d002f7a57be04915b7b0b3dcb985f550 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap complete.
I20240503 13:59:55.509516 13842 ts_tablet_manager.cc:1367] T d002f7a57be04915b7b0b3dcb985f550 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.004s
I20240503 13:59:55.509967 13842 raft_consensus.cc:348] T d002f7a57be04915b7b0b3dcb985f550 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:55.510094 13842 raft_consensus.cc:729] T d002f7a57be04915b7b0b3dcb985f550 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: LEARNER
I20240503 13:59:55.510198 13842 consensus_queue.cc:260] T d002f7a57be04915b7b0b3dcb985f550 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:55.510509 13842 ts_tablet_manager.cc:1398] T d002f7a57be04915b7b0b3dcb985f550 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:55.511765 6942 tablet_copy_service.cc:342] P f58065db70b64ef8b1dc499aa6ae4460: Request end of tablet copy session 4a85e492c8154e9881a9cc2a105d7e64-d002f7a57be04915b7b0b3dcb985f550 received from {username='slave'} at 127.0.99.131:38291
I20240503 13:59:55.511850 6942 tablet_copy_service.cc:433] P f58065db70b64ef8b1dc499aa6ae4460: ending tablet copy session 4a85e492c8154e9881a9cc2a105d7e64-d002f7a57be04915b7b0b3dcb985f550 on tablet d002f7a57be04915b7b0b3dcb985f550 with peer 4a85e492c8154e9881a9cc2a105d7e64
I20240503 13:59:55.524736 6922 raft_consensus.cc:1203] T aecb348d4b3b4500a587c92a7fefab4e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:55.536840 6657 raft_consensus.cc:1203] T 6ee85f4483004c1f9499203c7323706d P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:55.585562 13855 ts_tablet_manager.cc:902] T 218b25613baf48319576e433c2ac4534 P ab1d21935d6049888eb0ceccc3c9d735: Initiating tablet copy from peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:55.586067 13855 tablet_copy_client.cc:308] T 218b25613baf48319576e433c2ac4534 P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43531
I20240503 13:59:55.586663 6942 tablet_copy_service.cc:140] P f58065db70b64ef8b1dc499aa6ae4460: Received BeginTabletCopySession request for tablet 218b25613baf48319576e433c2ac4534 from peer ab1d21935d6049888eb0ceccc3c9d735 ({username='slave'} at 127.0.99.135:34893)
I20240503 13:59:55.586755 6942 tablet_copy_service.cc:161] P f58065db70b64ef8b1dc499aa6ae4460: Beginning new tablet copy session on tablet 218b25613baf48319576e433c2ac4534 from peer ab1d21935d6049888eb0ceccc3c9d735 at {username='slave'} at 127.0.99.135:34893: session id = ab1d21935d6049888eb0ceccc3c9d735-218b25613baf48319576e433c2ac4534
I20240503 13:59:55.587817 6942 tablet_copy_source_session.cc:205] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:55.588755 13855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 218b25613baf48319576e433c2ac4534. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:55.590981 13855 tablet_copy_client.cc:783] T 218b25613baf48319576e433c2ac4534 P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:55.591135 13855 tablet_copy_client.cc:647] T 218b25613baf48319576e433c2ac4534 P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:55.592111 13855 tablet_copy_client.cc:523] T 218b25613baf48319576e433c2ac4534 P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:55.593515 13855 tablet_bootstrap.cc:492] T 218b25613baf48319576e433c2ac4534 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:55.596941 13855 tablet_bootstrap.cc:492] T 218b25613baf48319576e433c2ac4534 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:55.597195 13855 tablet_bootstrap.cc:492] T 218b25613baf48319576e433c2ac4534 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap complete.
I20240503 13:59:55.597347 13855 ts_tablet_manager.cc:1367] T 218b25613baf48319576e433c2ac4534 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240503 13:59:55.597887 13855 raft_consensus.cc:348] T 218b25613baf48319576e433c2ac4534 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:55.598006 13855 raft_consensus.cc:729] T 218b25613baf48319576e433c2ac4534 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: LEARNER
I20240503 13:59:55.598109 13855 consensus_queue.cc:260] T 218b25613baf48319576e433c2ac4534 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:55.598404 13855 ts_tablet_manager.cc:1398] T 218b25613baf48319576e433c2ac4534 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 13:59:55.598874 6942 tablet_copy_service.cc:342] P f58065db70b64ef8b1dc499aa6ae4460: Request end of tablet copy session ab1d21935d6049888eb0ceccc3c9d735-218b25613baf48319576e433c2ac4534 received from {username='slave'} at 127.0.99.135:34893
I20240503 13:59:55.598945 6942 tablet_copy_service.cc:433] P f58065db70b64ef8b1dc499aa6ae4460: ending tablet copy session ab1d21935d6049888eb0ceccc3c9d735-218b25613baf48319576e433c2ac4534 on tablet 218b25613baf48319576e433c2ac4534 with peer ab1d21935d6049888eb0ceccc3c9d735
I20240503 13:59:55.631645 6788 consensus_queue.cc:237] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:55.632951 6528 raft_consensus.cc:1261] T 6ba45113776347c3adeae091c95427df P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:55.632973 9342 raft_consensus.cc:1261] T 6ba45113776347c3adeae091c95427df P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:55.633237 6727 consensus_peers.cc:485] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 -> Peer 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): Couldn't send request to peer 2145d15ab2d74c0287839266f1411a43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ba45113776347c3adeae091c95427df. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:55.633355 13276 consensus_queue.cc:1035] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:55.633711 13116 consensus_queue.cc:1035] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:55.636268 13276 raft_consensus.cc:2932] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:55.638056 9342 raft_consensus.cc:2932] T 6ba45113776347c3adeae091c95427df P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:55.640229 6528 raft_consensus.cc:2932] T 6ba45113776347c3adeae091c95427df P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:55.640200 6387 catalog_manager.cc:5484] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:55.645799 9079 raft_consensus.cc:1203] T 400c54bdecfa44e08c7eca046bcd2ffc P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:55.668673 9079 raft_consensus.cc:1203] T 01551845458048e48df9901947b4eec6 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:55.669806 9079 consensus_queue.cc:237] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 13:59:55.671105 6922 raft_consensus.cc:1261] T 64135eb6ef574c95b3677e32d66748a4 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:59:55.671169 9342 raft_consensus.cc:1261] T 64135eb6ef574c95b3677e32d66748a4 P 0d77752181e944698e809e325e5937e6 [term 2 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:59:55.671767 13280 consensus_queue.cc:1035] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:55.671985 12372 consensus_queue.cc:1035] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:55.673689 12372 raft_consensus.cc:2932] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:55.676060 6922 raft_consensus.cc:2932] T 64135eb6ef574c95b3677e32d66748a4 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:55.677495 9342 raft_consensus.cc:2932] T 64135eb6ef574c95b3677e32d66748a4 P 0d77752181e944698e809e325e5937e6 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:55.677641 6381 catalog_manager.cc:5484] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index 3 to 5, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:55.678309 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 64135eb6ef574c95b3677e32d66748a4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:55.681385 6769 tablet_service.cc:1514] Processing DeleteTablet for tablet 64135eb6ef574c95b3677e32d66748a4 with delete_type TABLET_DATA_TOMBSTONED (TS 4a85e492c8154e9881a9cc2a105d7e64 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:55.681742 14016 tablet_replica.cc:330] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:55.681890 14016 raft_consensus.cc:2229] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:55.682004 14016 raft_consensus.cc:2258] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.683276 14016 ts_tablet_manager.cc:1875] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:55.685704 14016 ts_tablet_manager.cc:1888] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 13:59:55.685823 14016 log.cc:1196] T 64135eb6ef574c95b3677e32d66748a4 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/64135eb6ef574c95b3677e32d66748a4
I20240503 13:59:55.686478 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet 64135eb6ef574c95b3677e32d66748a4 (table rebalancer_extra_table_20 [id=c8e9e03cbc904134b1bfd2851cf3b5a7]) successfully deleted
I20240503 13:59:55.783617 6528 raft_consensus.cc:1203] T 44eddd53c3f348a9a1117a20473cbb65 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:55.795491 8946 consensus_queue.cc:237] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:55.796749 9342 raft_consensus.cc:1261] T 9f0daf41253f4678b158e3ccbb7b0c98 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:55.796897 9205 raft_consensus.cc:1261] T 9f0daf41253f4678b158e3ccbb7b0c98 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:55.797190 8882 consensus_peers.cc:485] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9f0daf41253f4678b158e3ccbb7b0c98. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:55.797245 13268 consensus_queue.cc:1035] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:55.797412 12367 consensus_queue.cc:1035] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:55.799068 11750 raft_consensus.cc:2932] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:55.799507 9342 raft_consensus.cc:2932] T 9f0daf41253f4678b158e3ccbb7b0c98 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:55.802402 9205 raft_consensus.cc:2932] T 9f0daf41253f4678b158e3ccbb7b0c98 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:55.802649 6379 catalog_manager.cc:5484] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:55.807811 9342 raft_consensus.cc:1203] T e106b1432baf423493c95fcce63a55da P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:55.865796 12864 raft_consensus.cc:1050] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4: attempting to promote NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 to VOTER
I20240503 13:59:55.866235 12864 consensus_queue.cc:237] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
I20240503 13:59:55.867532 9342 raft_consensus.cc:1261] T f1291c3100bb4533891a1a7a3357d363 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:55.867718 8946 raft_consensus.cc:1261] T f1291c3100bb4533891a1a7a3357d363 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:55.867888 6528 raft_consensus.cc:1261] T f1291c3100bb4533891a1a7a3357d363 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:55.868186 14146 consensus_queue.cc:1035] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:55.868510 12372 consensus_queue.cc:1035] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:55.869192 12372 consensus_queue.cc:1035] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:55.871793 12372 raft_consensus.cc:2932] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:55.871949 6922 raft_consensus.cc:1203] T 61dfb1d40ee444368470a0dafc3179a4 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:55.873906 6528 raft_consensus.cc:2932] T f1291c3100bb4533891a1a7a3357d363 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:55.874594 8946 raft_consensus.cc:2932] T f1291c3100bb4533891a1a7a3357d363 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:55.874778 9342 raft_consensus.cc:1203] T ef19b1e5f15e49448c1b2919a24090d3 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:55.878906 6387 catalog_manager.cc:5484] T f1291c3100bb4533891a1a7a3357d363 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index 2 to 3, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:55.881340 9342 raft_consensus.cc:2932] T f1291c3100bb4533891a1a7a3357d363 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:55.885116 9079 consensus_queue.cc:237] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
I20240503 13:59:55.887251 8946 raft_consensus.cc:1261] T f1291c3100bb4533891a1a7a3357d363 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:55.887670 14146 consensus_queue.cc:1035] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:55.888738 6528 raft_consensus.cc:1261] T f1291c3100bb4533891a1a7a3357d363 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:55.889181 14146 consensus_queue.cc:1035] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:55.890282 12372 raft_consensus.cc:2932] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:55.891175 6528 raft_consensus.cc:2932] T f1291c3100bb4533891a1a7a3357d363 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:55.893573 6381 catalog_manager.cc:5484] T f1291c3100bb4533891a1a7a3357d363 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:55.896385 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f1291c3100bb4533891a1a7a3357d363 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:55.896409 8947 raft_consensus.cc:2932] T f1291c3100bb4533891a1a7a3357d363 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:55.896690 9321 tablet_service.cc:1514] Processing DeleteTablet for tablet f1291c3100bb4533891a1a7a3357d363 with delete_type TABLET_DATA_TOMBSTONED (TS 0d77752181e944698e809e325e5937e6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:55.897085 14160 tablet_replica.cc:330] T f1291c3100bb4533891a1a7a3357d363 P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:55.897223 14160 raft_consensus.cc:2229] T f1291c3100bb4533891a1a7a3357d363 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:55.897380 14160 raft_consensus.cc:2258] T f1291c3100bb4533891a1a7a3357d363 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.898053 14160 ts_tablet_manager.cc:1875] T f1291c3100bb4533891a1a7a3357d363 P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:55.902546 14160 ts_tablet_manager.cc:1888] T f1291c3100bb4533891a1a7a3357d363 P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:55.902616 14160 log.cc:1196] T f1291c3100bb4533891a1a7a3357d363 P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/f1291c3100bb4533891a1a7a3357d363
I20240503 13:59:55.903138 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet f1291c3100bb4533891a1a7a3357d363 (table rebalancer_extra_table_44 [id=a8e55189edbc4c54bebc8aac90a27231]) successfully deleted
I20240503 13:59:55.906536 13080 raft_consensus.cc:1050] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6: attempting to promote NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 to VOTER
I20240503 13:59:55.906975 13080 consensus_queue.cc:237] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 13:59:55.908280 8947 raft_consensus.cc:1261] T 0b397d0460c94438a773be184312b60f P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:55.908713 14167 consensus_queue.cc:1035] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:55.908927 6922 raft_consensus.cc:1261] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:55.909404 9079 raft_consensus.cc:1261] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:55.909991 14167 consensus_queue.cc:1035] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:55.912062 14167 consensus_queue.cc:1035] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:55.914192 13080 raft_consensus.cc:2932] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:55.914609 9079 raft_consensus.cc:2932] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:55.916833 8947 raft_consensus.cc:2932] T 0b397d0460c94438a773be184312b60f P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:55.918028 6379 catalog_manager.cc:5484] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:55.918563 6791 raft_consensus.cc:1203] T 4a1398d983c54a7498257f766052e8e9 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:55.922412 6920 raft_consensus.cc:2932] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:55.923732 9342 consensus_queue.cc:237] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 13:59:55.925743 6920 raft_consensus.cc:1261] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:55.925834 13325 raft_consensus.cc:1050] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43: attempting to promote NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 to VOTER
I20240503 13:59:55.926136 13080 consensus_queue.cc:1035] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:55.926263 13325 consensus_queue.cc:237] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 13:59:55.927265 9079 raft_consensus.cc:1261] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:55.927918 9205 raft_consensus.cc:1261] T d588a730a0c94d4f9446049e537c6727 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:55.927940 6922 raft_consensus.cc:1261] T d588a730a0c94d4f9446049e537c6727 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:55.928381 14177 consensus_queue.cc:1035] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:55.928563 13080 consensus_queue.cc:1035] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:55.928654 13575 consensus_queue.cc:1035] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:55.929922 9079 raft_consensus.cc:1261] T d588a730a0c94d4f9446049e537c6727 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:55.930274 13575 consensus_queue.cc:1035] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:55.930290 11378 raft_consensus.cc:2932] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:55.930713 6922 raft_consensus.cc:2932] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:55.932029 6374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0b397d0460c94438a773be184312b60f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:55.933775 6385 catalog_manager.cc:5484] T 0b397d0460c94438a773be184312b60f P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:55.934417 6791 raft_consensus.cc:1203] T d002f7a57be04915b7b0b3dcb985f550 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20240503 13:59:55.937127 6791 raft_consensus.cc:1203] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:55.937546 8926 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b397d0460c94438a773be184312b60f with delete_type TABLET_DATA_TOMBSTONED (TS 06ca8dbc004c42b99faa527a4e6f9545 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:55.938057 14181 tablet_replica.cc:330] T 0b397d0460c94438a773be184312b60f P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:55.938243 14181 raft_consensus.cc:2229] T 0b397d0460c94438a773be184312b60f P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:55.938470 14181 raft_consensus.cc:2258] T 0b397d0460c94438a773be184312b60f P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.941362 9081 raft_consensus.cc:2932] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:55.941954 14181 ts_tablet_manager.cc:1875] T 0b397d0460c94438a773be184312b60f P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:55.942394 9210 raft_consensus.cc:1203] T da69140936424a31b66c87e2ad4092cb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:55.944515 14181 ts_tablet_manager.cc:1888] T 0b397d0460c94438a773be184312b60f P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:55.944597 14181 log.cc:1196] T 0b397d0460c94438a773be184312b60f P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/0b397d0460c94438a773be184312b60f
I20240503 13:59:55.945168 14177 raft_consensus.cc:2932] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:55.945618 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 0b397d0460c94438a773be184312b60f (table rebalancer_extra_table_12 [id=3fa9362ac0d94323a9cd7a56209c3e07]) successfully deleted
I20240503 13:59:55.945894 9079 raft_consensus.cc:2932] T d588a730a0c94d4f9446049e537c6727 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:55.948632 6922 raft_consensus.cc:2932] T d588a730a0c94d4f9446049e537c6727 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:55.949272 9205 raft_consensus.cc:2932] T d588a730a0c94d4f9446049e537c6727 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:55.950182 6379 catalog_manager.cc:5484] T d588a730a0c94d4f9446049e537c6727 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:55.957687 6657 consensus_queue.cc:237] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 13:59:55.958676 9205 raft_consensus.cc:1261] T d588a730a0c94d4f9446049e537c6727 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:55.958691 6922 raft_consensus.cc:1261] T d588a730a0c94d4f9446049e537c6727 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:55.959007 13575 consensus_queue.cc:1035] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:55.959194 14177 consensus_queue.cc:1035] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:55.960621 13325 raft_consensus.cc:2932] T d588a730a0c94d4f9446049e537c6727 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:55.962374 9205 raft_consensus.cc:2932] T d588a730a0c94d4f9446049e537c6727 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:55.961654 6922 raft_consensus.cc:2932] T d588a730a0c94d4f9446049e537c6727 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:55.966097 6374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d588a730a0c94d4f9446049e537c6727 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:55.967530 6387 catalog_manager.cc:5484] T d588a730a0c94d4f9446049e537c6727 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:55.971935 9060 tablet_service.cc:1514] Processing DeleteTablet for tablet d588a730a0c94d4f9446049e537c6727 with delete_type TABLET_DATA_TOMBSTONED (TS 8d574fb4a89d4eea947c859470d44dd4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:55.973237 14192 tablet_replica.cc:330] T d588a730a0c94d4f9446049e537c6727 P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:55.973413 14192 raft_consensus.cc:2229] T d588a730a0c94d4f9446049e537c6727 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:55.973585 14192 raft_consensus.cc:2258] T d588a730a0c94d4f9446049e537c6727 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.975853 14192 ts_tablet_manager.cc:1875] T d588a730a0c94d4f9446049e537c6727 P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:55.978241 14192 ts_tablet_manager.cc:1888] T d588a730a0c94d4f9446049e537c6727 P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:55.978384 14192 log.cc:1196] T d588a730a0c94d4f9446049e537c6727 P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/d588a730a0c94d4f9446049e537c6727
I20240503 13:59:55.978940 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet d588a730a0c94d4f9446049e537c6727 (table rebalancer_extra_table_44 [id=a8e55189edbc4c54bebc8aac90a27231]) successfully deleted
I20240503 13:59:56.040871 9342 raft_consensus.cc:1203] T ea45c42d7fed4171a1b726b0684d2fa5 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:56.061079 12508 raft_consensus.cc:1050] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da: attempting to promote NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 to VOTER
I20240503 13:59:56.061518 12508 consensus_queue.cc:237] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 13:59:56.062701 6922 raft_consensus.cc:1261] T aecb348d4b3b4500a587c92a7fefab4e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:56.062909 9342 raft_consensus.cc:1261] T aecb348d4b3b4500a587c92a7fefab4e P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:56.062937 9079 raft_consensus.cc:1261] T aecb348d4b3b4500a587c92a7fefab4e P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:56.063082 13157 consensus_queue.cc:1035] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:56.063381 14214 consensus_queue.cc:1035] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:56.063561 13157 consensus_queue.cc:1035] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:56.065348 13157 raft_consensus.cc:2932] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:56.065816 9079 raft_consensus.cc:2932] T aecb348d4b3b4500a587c92a7fefab4e P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:56.065950 9342 raft_consensus.cc:2932] T aecb348d4b3b4500a587c92a7fefab4e P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:56.068426 6922 raft_consensus.cc:2932] T aecb348d4b3b4500a587c92a7fefab4e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:56.069473 6387 catalog_manager.cc:5484] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:56.073179 6528 consensus_queue.cc:237] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 13:59:56.074091 6922 raft_consensus.cc:1261] T aecb348d4b3b4500a587c92a7fefab4e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.074229 9342 raft_consensus.cc:1261] T aecb348d4b3b4500a587c92a7fefab4e P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.074517 12508 consensus_queue.cc:1035] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.074739 14214 consensus_queue.cc:1035] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.076239 13902 raft_consensus.cc:2932] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:56.076625 6922 raft_consensus.cc:2932] T aecb348d4b3b4500a587c92a7fefab4e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:56.078292 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet aecb348d4b3b4500a587c92a7fefab4e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:56.079408 9342 raft_consensus.cc:2932] T aecb348d4b3b4500a587c92a7fefab4e P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:56.079551 6384 catalog_manager.cc:5484] T aecb348d4b3b4500a587c92a7fefab4e P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:56.082577 9060 tablet_service.cc:1514] Processing DeleteTablet for tablet aecb348d4b3b4500a587c92a7fefab4e with delete_type TABLET_DATA_TOMBSTONED (TS 8d574fb4a89d4eea947c859470d44dd4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:56.082769 14192 tablet_replica.cc:330] T aecb348d4b3b4500a587c92a7fefab4e P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:56.084810 14192 raft_consensus.cc:2229] T aecb348d4b3b4500a587c92a7fefab4e P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:56.084967 14192 raft_consensus.cc:2258] T aecb348d4b3b4500a587c92a7fefab4e P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:56.085445 14192 ts_tablet_manager.cc:1875] T aecb348d4b3b4500a587c92a7fefab4e P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:56.087114 14192 ts_tablet_manager.cc:1888] T aecb348d4b3b4500a587c92a7fefab4e P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:56.087200 14192 log.cc:1196] T aecb348d4b3b4500a587c92a7fefab4e P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/aecb348d4b3b4500a587c92a7fefab4e
I20240503 13:59:56.087765 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet aecb348d4b3b4500a587c92a7fefab4e (table rebalancer_extra_table_24 [id=595e2f47ca074ec4bd181e870ba60ff9]) successfully deleted
I20240503 13:59:56.126204 13159 raft_consensus.cc:1050] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460: attempting to promote NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 to VOTER
I20240503 13:59:56.126617 13159 consensus_queue.cc:237] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 13:59:56.128196 9079 raft_consensus.cc:1261] T 400c54bdecfa44e08c7eca046bcd2ffc P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:56.128222 6528 raft_consensus.cc:1261] T 400c54bdecfa44e08c7eca046bcd2ffc P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:56.128670 14219 consensus_queue.cc:1035] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:56.128868 13370 consensus_queue.cc:1035] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:56.129403 6657 raft_consensus.cc:1261] T 400c54bdecfa44e08c7eca046bcd2ffc P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:56.129894 14219 consensus_queue.cc:1035] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:56.131021 13370 raft_consensus.cc:2932] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:56.131556 6528 raft_consensus.cc:2932] T 400c54bdecfa44e08c7eca046bcd2ffc P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:56.133561 9079 raft_consensus.cc:2932] T 400c54bdecfa44e08c7eca046bcd2ffc P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:56.134451 6657 raft_consensus.cc:2932] T 400c54bdecfa44e08c7eca046bcd2ffc P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:56.135203 6380 catalog_manager.cc:5484] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 4 to 5, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:56.139036 11378 raft_consensus.cc:1050] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6: attempting to promote NON_VOTER 2145d15ab2d74c0287839266f1411a43 to VOTER
I20240503 13:59:56.140602 11378 consensus_queue.cc:237] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 13:59:56.142136 9210 raft_consensus.cc:1261] T 6ee85f4483004c1f9499203c7323706d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:56.142433 6920 raft_consensus.cc:1261] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:56.142587 11167 consensus_queue.cc:1035] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:56.143066 9210 raft_consensus.cc:1203] T 218b25613baf48319576e433c2ac4534 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:56.143373 14167 consensus_queue.cc:1035] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:56.144295 6922 consensus_queue.cc:237] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 13:59:56.145309 6659 raft_consensus.cc:1261] T 6ee85f4483004c1f9499203c7323706d P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:56.145685 11167 consensus_queue.cc:1035] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:56.146605 9079 raft_consensus.cc:1261] T 400c54bdecfa44e08c7eca046bcd2ffc P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:56.147454 11378 raft_consensus.cc:2932] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:56.148648 9205 raft_consensus.cc:2932] T 6ee85f4483004c1f9499203c7323706d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:56.148806 6657 raft_consensus.cc:1261] T 400c54bdecfa44e08c7eca046bcd2ffc P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:56.149471 13159 consensus_queue.cc:1035] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:56.151633 6381 catalog_manager.cc:5484] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:56.152277 14219 consensus_queue.cc:1035] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:56.152385 6920 raft_consensus.cc:2932] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:56.152740 12748 raft_consensus.cc:2932] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:56.153952 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 400c54bdecfa44e08c7eca046bcd2ffc with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:56.154361 9079 raft_consensus.cc:2932] T 400c54bdecfa44e08c7eca046bcd2ffc P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:56.157415 6387 catalog_manager.cc:5484] T 400c54bdecfa44e08c7eca046bcd2ffc P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 5 to 6, VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:56.158056 6657 raft_consensus.cc:2932] T 400c54bdecfa44e08c7eca046bcd2ffc P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:56.159683 6507 tablet_service.cc:1514] Processing DeleteTablet for tablet 400c54bdecfa44e08c7eca046bcd2ffc with delete_type TABLET_DATA_TOMBSTONED (TS 5f46e19644fd4f4b825c86ba6af768da not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:56.160007 14227 tablet_replica.cc:330] T 400c54bdecfa44e08c7eca046bcd2ffc P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:56.160154 14227 raft_consensus.cc:2229] T 400c54bdecfa44e08c7eca046bcd2ffc P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:56.160315 14227 raft_consensus.cc:2258] T 400c54bdecfa44e08c7eca046bcd2ffc P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:56.161903 6659 raft_consensus.cc:2932] T 6ee85f4483004c1f9499203c7323706d P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:56.162175 14227 ts_tablet_manager.cc:1875] T 400c54bdecfa44e08c7eca046bcd2ffc P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:56.164530 14227 ts_tablet_manager.cc:1888] T 400c54bdecfa44e08c7eca046bcd2ffc P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:56.165092 9342 consensus_queue.cc:237] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 13:59:56.165694 14227 log.cc:1196] T 400c54bdecfa44e08c7eca046bcd2ffc P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/400c54bdecfa44e08c7eca046bcd2ffc
I20240503 13:59:56.166208 6659 raft_consensus.cc:1261] T 6ee85f4483004c1f9499203c7323706d P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.166258 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 400c54bdecfa44e08c7eca046bcd2ffc (table rebalance_test_table_15 [id=65cfc473fc21464c991e7a62f281b04e]) successfully deleted
I20240503 13:59:56.166359 6920 raft_consensus.cc:1261] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.166630 14167 consensus_queue.cc:1035] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.166813 11378 consensus_queue.cc:1035] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.168123 14167 raft_consensus.cc:2932] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:56.168705 6659 raft_consensus.cc:2932] T 6ee85f4483004c1f9499203c7323706d P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:56.169562 6374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6ee85f4483004c1f9499203c7323706d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:56.169970 6920 raft_consensus.cc:2932] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:56.171917 6380 catalog_manager.cc:5484] T 6ee85f4483004c1f9499203c7323706d P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:56.174117 9192 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ee85f4483004c1f9499203c7323706d with delete_type TABLET_DATA_TOMBSTONED (TS ab1d21935d6049888eb0ceccc3c9d735 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57960
I20240503 13:59:56.174450 14228 tablet_replica.cc:330] T 6ee85f4483004c1f9499203c7323706d P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 13:59:56.174568 14228 raft_consensus.cc:2229] T 6ee85f4483004c1f9499203c7323706d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:56.174726 14228 raft_consensus.cc:2258] T 6ee85f4483004c1f9499203c7323706d P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:56.175182 14228 ts_tablet_manager.cc:1875] T 6ee85f4483004c1f9499203c7323706d P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:56.177651 14228 ts_tablet_manager.cc:1888] T 6ee85f4483004c1f9499203c7323706d P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:56.177708 14228 log.cc:1196] T 6ee85f4483004c1f9499203c7323706d P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/6ee85f4483004c1f9499203c7323706d
I20240503 13:59:56.178097 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet 6ee85f4483004c1f9499203c7323706d (table rebalancer_extra_table_32 [id=69fc27437d404cf4a64ea4f3de479bfc]) successfully deleted
I20240503 13:59:56.192386 13276 raft_consensus.cc:1050] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64: attempting to promote NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 to VOTER
I20240503 13:59:56.192813 13276 consensus_queue.cc:237] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 13:59:56.194075 8947 raft_consensus.cc:1261] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:56.194227 6526 raft_consensus.cc:1261] T 01551845458048e48df9901947b4eec6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:56.194427 14230 consensus_queue.cc:1035] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:56.194491 9079 raft_consensus.cc:1261] T 01551845458048e48df9901947b4eec6 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:56.194679 13116 consensus_queue.cc:1035] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:56.194820 13276 consensus_queue.cc:1035] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:56.196662 13276 raft_consensus.cc:2932] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:56.197139 6526 raft_consensus.cc:2932] T 01551845458048e48df9901947b4eec6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:56.198760 8947 raft_consensus.cc:2932] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:56.200335 6387 catalog_manager.cc:5484] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:56.202222 9079 raft_consensus.cc:2932] T 01551845458048e48df9901947b4eec6 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:56.205072 14234 ts_tablet_manager.cc:902] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43: Initiating tablet copy from peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:56.205731 14234 tablet_copy_client.cc:308] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:46365
I20240503 13:59:56.206532 6811 tablet_copy_service.cc:140] P 4a85e492c8154e9881a9cc2a105d7e64: Received BeginTabletCopySession request for tablet 6ba45113776347c3adeae091c95427df from peer 2145d15ab2d74c0287839266f1411a43 ({username='slave'} at 127.0.99.130:47541)
I20240503 13:59:56.206622 6811 tablet_copy_service.cc:161] P 4a85e492c8154e9881a9cc2a105d7e64: Beginning new tablet copy session on tablet 6ba45113776347c3adeae091c95427df from peer 2145d15ab2d74c0287839266f1411a43 at {username='slave'} at 127.0.99.130:47541: session id = 2145d15ab2d74c0287839266f1411a43-6ba45113776347c3adeae091c95427df
I20240503 13:59:56.207530 6791 consensus_queue.cc:237] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 13:59:56.208786 6811 tablet_copy_source_session.cc:205] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:56.209079 9079 raft_consensus.cc:1261] T 01551845458048e48df9901947b4eec6 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.209375 8947 raft_consensus.cc:1261] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.209826 14234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ba45113776347c3adeae091c95427df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:56.210690 13276 consensus_queue.cc:1035] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.210834 14230 consensus_queue.cc:1035] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.212484 14234 tablet_copy_client.cc:783] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:56.212651 14234 tablet_copy_client.cc:647] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:56.212644 13276 raft_consensus.cc:2932] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:56.213572 9079 raft_consensus.cc:2932] T 01551845458048e48df9901947b4eec6 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:56.214162 6372 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 01551845458048e48df9901947b4eec6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:56.214587 8947 raft_consensus.cc:2932] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:56.215953 14234 tablet_copy_client.cc:523] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:56.216466 6380 catalog_manager.cc:5484] T 01551845458048e48df9901947b4eec6 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index 3 to 4, VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:56.219633 6507 tablet_service.cc:1514] Processing DeleteTablet for tablet 01551845458048e48df9901947b4eec6 with delete_type TABLET_DATA_TOMBSTONED (TS 5f46e19644fd4f4b825c86ba6af768da not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:56.219880 14227 tablet_replica.cc:330] T 01551845458048e48df9901947b4eec6 P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:56.220028 14227 raft_consensus.cc:2229] T 01551845458048e48df9901947b4eec6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:56.220167 14227 raft_consensus.cc:2258] T 01551845458048e48df9901947b4eec6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:56.221040 14227 ts_tablet_manager.cc:1875] T 01551845458048e48df9901947b4eec6 P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:56.223068 14227 ts_tablet_manager.cc:1888] T 01551845458048e48df9901947b4eec6 P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:56.223116 14227 log.cc:1196] T 01551845458048e48df9901947b4eec6 P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/01551845458048e48df9901947b4eec6
I20240503 13:59:56.223994 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 01551845458048e48df9901947b4eec6 (table rebalancer_extra_table_72 [id=fb05f2e903bb419db0bfe7d54f35b083]) successfully deleted
I20240503 13:59:56.224170 14234 tablet_bootstrap.cc:492] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:56.227627 14234 tablet_bootstrap.cc:492] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:56.227900 14234 tablet_bootstrap.cc:492] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43: Bootstrap complete.
I20240503 13:59:56.228060 14234 ts_tablet_manager.cc:1367] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 13:59:56.228567 14234 raft_consensus.cc:348] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:56.228778 14234 raft_consensus.cc:729] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: LEARNER
I20240503 13:59:56.228945 14234 consensus_queue.cc:260] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:56.230295 14234 ts_tablet_manager.cc:1398] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:56.230928 6811 tablet_copy_service.cc:342] P 4a85e492c8154e9881a9cc2a105d7e64: Request end of tablet copy session 2145d15ab2d74c0287839266f1411a43-6ba45113776347c3adeae091c95427df received from {username='slave'} at 127.0.99.130:47541
I20240503 13:59:56.231010 6811 tablet_copy_service.cc:433] P 4a85e492c8154e9881a9cc2a105d7e64: ending tablet copy session 2145d15ab2d74c0287839266f1411a43-6ba45113776347c3adeae091c95427df on tablet 6ba45113776347c3adeae091c95427df with peer 2145d15ab2d74c0287839266f1411a43
I20240503 13:59:56.292238 13902 raft_consensus.cc:1050] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da: attempting to promote NON_VOTER 0d77752181e944698e809e325e5937e6 to VOTER
I20240503 13:59:56.292683 13902 consensus_queue.cc:237] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 13:59:56.294248 6788 raft_consensus.cc:1261] T e106b1432baf423493c95fcce63a55da P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:56.294744 14214 consensus_queue.cc:1035] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:56.295719 9342 raft_consensus.cc:1261] T e106b1432baf423493c95fcce63a55da P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:56.295735 6659 raft_consensus.cc:1261] T e106b1432baf423493c95fcce63a55da P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:56.296747 14214 consensus_queue.cc:1035] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:56.296985 13157 consensus_queue.cc:1035] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:56.298909 13157 raft_consensus.cc:2932] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:56.299503 6657 raft_consensus.cc:2932] T e106b1432baf423493c95fcce63a55da P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:56.299926 9342 raft_consensus.cc:2932] T e106b1432baf423493c95fcce63a55da P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:56.301098 6788 raft_consensus.cc:2932] T e106b1432baf423493c95fcce63a55da P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:56.304059 6384 catalog_manager.cc:5484] T e106b1432baf423493c95fcce63a55da P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:56.315701 6528 consensus_queue.cc:237] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 13:59:56.316998 9342 raft_consensus.cc:1261] T e106b1432baf423493c95fcce63a55da P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:56.317466 13902 consensus_queue.cc:1035] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:56.317745 6657 raft_consensus.cc:1261] T e106b1432baf423493c95fcce63a55da P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:56.318162 13902 consensus_queue.cc:1035] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:56.319440 14214 raft_consensus.cc:2932] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:56.320158 9342 raft_consensus.cc:2932] T e106b1432baf423493c95fcce63a55da P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:56.321161 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e106b1432baf423493c95fcce63a55da with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:56.323927 6657 raft_consensus.cc:2932] T e106b1432baf423493c95fcce63a55da P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:56.324246 6380 catalog_manager.cc:5484] T e106b1432baf423493c95fcce63a55da P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index 5 to 6, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:56.327948 6769 tablet_service.cc:1514] Processing DeleteTablet for tablet e106b1432baf423493c95fcce63a55da with delete_type TABLET_DATA_TOMBSTONED (TS 4a85e492c8154e9881a9cc2a105d7e64 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:56.335456 14314 tablet_replica.cc:330] T e106b1432baf423493c95fcce63a55da P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:56.335610 14314 raft_consensus.cc:2229] T e106b1432baf423493c95fcce63a55da P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:56.335769 14314 raft_consensus.cc:2258] T e106b1432baf423493c95fcce63a55da P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:56.337141 11378 raft_consensus.cc:1050] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6: attempting to promote NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 to VOTER
I20240503 13:59:56.337245 14314 ts_tablet_manager.cc:1875] T e106b1432baf423493c95fcce63a55da P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:56.337553 11378 consensus_queue.cc:237] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 13:59:56.338846 6657 raft_consensus.cc:1261] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:56.339190 14167 consensus_queue.cc:1035] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:56.339677 14314 ts_tablet_manager.cc:1888] T e106b1432baf423493c95fcce63a55da P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:56.339780 14314 log.cc:1196] T e106b1432baf423493c95fcce63a55da P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/e106b1432baf423493c95fcce63a55da
I20240503 13:59:56.339821 6788 raft_consensus.cc:1261] T 4a1398d983c54a7498257f766052e8e9 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:56.339807 8947 raft_consensus.cc:1261] T 4a1398d983c54a7498257f766052e8e9 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:56.340168 14167 consensus_queue.cc:1035] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:56.340438 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet e106b1432baf423493c95fcce63a55da (table rebalance_test_table_3 [id=0ea567ca26f349a788891ba7d2d3751a]) successfully deleted
I20240503 13:59:56.340821 14167 consensus_queue.cc:1035] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:56.342473 13080 raft_consensus.cc:2932] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:56.343729 6788 raft_consensus.cc:2932] T 4a1398d983c54a7498257f766052e8e9 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:56.344221 6657 raft_consensus.cc:2932] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:56.345319 8947 raft_consensus.cc:2932] T 4a1398d983c54a7498257f766052e8e9 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:56.347786 6384 catalog_manager.cc:5484] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:56.351230 9342 consensus_queue.cc:237] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 13:59:56.353657 6659 raft_consensus.cc:1261] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.354496 14167 consensus_queue.cc:1035] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.356477 6788 raft_consensus.cc:1261] T 4a1398d983c54a7498257f766052e8e9 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.356508 13080 raft_consensus.cc:2932] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:56.357792 6374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4a1398d983c54a7498257f766052e8e9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:56.357915 11167 consensus_queue.cc:1035] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.358898 6659 raft_consensus.cc:2932] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:56.359189 6788 raft_consensus.cc:2932] T 4a1398d983c54a7498257f766052e8e9 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:56.361200 6387 catalog_manager.cc:5484] T 4a1398d983c54a7498257f766052e8e9 P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:56.363225 14324 ts_tablet_manager.cc:902] T 9f0daf41253f4678b158e3ccbb7b0c98 P 4a85e492c8154e9881a9cc2a105d7e64: Initiating tablet copy from peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:56.363574 12864 raft_consensus.cc:1050] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4: attempting to promote NON_VOTER 5f46e19644fd4f4b825c86ba6af768da to VOTER
I20240503 13:59:56.363801 14324 tablet_copy_client.cc:308] T 9f0daf41253f4678b158e3ccbb7b0c98 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:34547
I20240503 13:59:56.364004 12864 consensus_queue.cc:237] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 13:59:56.365134 8926 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a1398d983c54a7498257f766052e8e9 with delete_type TABLET_DATA_TOMBSTONED (TS 06ca8dbc004c42b99faa527a4e6f9545 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:56.365355 14181 tablet_replica.cc:330] T 4a1398d983c54a7498257f766052e8e9 P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:56.365492 14181 raft_consensus.cc:2229] T 4a1398d983c54a7498257f766052e8e9 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:56.365648 14181 raft_consensus.cc:2258] T 4a1398d983c54a7498257f766052e8e9 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:56.366066 8968 tablet_copy_service.cc:140] P 06ca8dbc004c42b99faa527a4e6f9545: Received BeginTabletCopySession request for tablet 9f0daf41253f4678b158e3ccbb7b0c98 from peer 4a85e492c8154e9881a9cc2a105d7e64 ({username='slave'} at 127.0.99.131:54275)
I20240503 13:59:56.366153 8968 tablet_copy_service.cc:161] P 06ca8dbc004c42b99faa527a4e6f9545: Beginning new tablet copy session on tablet 9f0daf41253f4678b158e3ccbb7b0c98 from peer 4a85e492c8154e9881a9cc2a105d7e64 at {username='slave'} at 127.0.99.131:54275: session id = 4a85e492c8154e9881a9cc2a105d7e64-9f0daf41253f4678b158e3ccbb7b0c98
I20240503 13:59:56.367151 8968 tablet_copy_source_session.cc:205] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:56.367494 14177 raft_consensus.cc:1050] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43: attempting to promote NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 to VOTER
I20240503 13:59:56.367902 14177 consensus_queue.cc:237] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 13:59:56.369412 14324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f0daf41253f4678b158e3ccbb7b0c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:56.369463 9342 raft_consensus.cc:1261] T 61dfb1d40ee444368470a0dafc3179a4 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:56.369599 6920 raft_consensus.cc:1261] T 61dfb1d40ee444368470a0dafc3179a4 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:56.369881 14335 consensus_queue.cc:1035] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:56.370194 14177 consensus_queue.cc:1035] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:56.370699 14181 ts_tablet_manager.cc:1875] T 4a1398d983c54a7498257f766052e8e9 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:56.371815 14324 tablet_copy_client.cc:783] T 9f0daf41253f4678b158e3ccbb7b0c98 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:56.371991 14324 tablet_copy_client.cc:647] T 9f0daf41253f4678b158e3ccbb7b0c98 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:56.372936 14181 ts_tablet_manager.cc:1888] T 4a1398d983c54a7498257f766052e8e9 P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:56.373005 14181 log.cc:1196] T 4a1398d983c54a7498257f766052e8e9 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/4a1398d983c54a7498257f766052e8e9
I20240503 13:59:56.373296 14324 tablet_copy_client.cc:523] T 9f0daf41253f4678b158e3ccbb7b0c98 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:56.373535 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 4a1398d983c54a7498257f766052e8e9 (table rebalancer_extra_table_68 [id=2d921adc27d94e8bae2f87cf4caf89d7]) successfully deleted
I20240503 13:59:56.373889 9210 raft_consensus.cc:1261] T 44eddd53c3f348a9a1117a20473cbb65 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:56.374369 9079 raft_consensus.cc:1261] T 61dfb1d40ee444368470a0dafc3179a4 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:56.374804 14335 consensus_queue.cc:1035] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:56.374904 14324 tablet_bootstrap.cc:492] T 9f0daf41253f4678b158e3ccbb7b0c98 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:56.374962 14146 consensus_queue.cc:1035] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:56.375803 6528 raft_consensus.cc:1261] T 44eddd53c3f348a9a1117a20473cbb65 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:56.376277 9342 raft_consensus.cc:1261] T 44eddd53c3f348a9a1117a20473cbb65 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:56.376660 14146 consensus_queue.cc:1035] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:56.378672 14146 consensus_queue.cc:1035] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:56.380225 14324 tablet_bootstrap.cc:492] T 9f0daf41253f4678b158e3ccbb7b0c98 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:56.380546 14324 tablet_bootstrap.cc:492] T 9f0daf41253f4678b158e3ccbb7b0c98 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap complete.
I20240503 13:59:56.380717 14324 ts_tablet_manager.cc:1367] T 9f0daf41253f4678b158e3ccbb7b0c98 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20240503 13:59:56.381143 14324 raft_consensus.cc:348] T 9f0daf41253f4678b158e3ccbb7b0c98 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:56.381258 14324 raft_consensus.cc:729] T 9f0daf41253f4678b158e3ccbb7b0c98 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: LEARNER
I20240503 13:59:56.381633 14324 consensus_queue.cc:260] T 9f0daf41253f4678b158e3ccbb7b0c98 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:56.382764 13325 raft_consensus.cc:2932] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:56.382899 13280 raft_consensus.cc:2932] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:56.383500 9210 raft_consensus.cc:2932] T 44eddd53c3f348a9a1117a20473cbb65 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:56.385403 9342 raft_consensus.cc:2932] T 61dfb1d40ee444368470a0dafc3179a4 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:56.385989 14324 ts_tablet_manager.cc:1398] T 9f0daf41253f4678b158e3ccbb7b0c98 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240503 13:59:56.387876 9079 raft_consensus.cc:2932] T 61dfb1d40ee444368470a0dafc3179a4 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:56.388974 9342 raft_consensus.cc:2932] T 44eddd53c3f348a9a1117a20473cbb65 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:56.389235 6380 catalog_manager.cc:5484] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:56.390668 6384 catalog_manager.cc:5484] T 44eddd53c3f348a9a1117a20473cbb65 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:56.392772 6526 raft_consensus.cc:2932] T 44eddd53c3f348a9a1117a20473cbb65 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:56.395073 8968 tablet_copy_service.cc:342] P 06ca8dbc004c42b99faa527a4e6f9545: Request end of tablet copy session 4a85e492c8154e9881a9cc2a105d7e64-9f0daf41253f4678b158e3ccbb7b0c98 received from {username='slave'} at 127.0.99.131:54275
I20240503 13:59:56.395161 8968 tablet_copy_service.cc:433] P 06ca8dbc004c42b99faa527a4e6f9545: ending tablet copy session 4a85e492c8154e9881a9cc2a105d7e64-9f0daf41253f4678b158e3ccbb7b0c98 on tablet 9f0daf41253f4678b158e3ccbb7b0c98 with peer 4a85e492c8154e9881a9cc2a105d7e64
I20240503 13:59:56.395757 14167 raft_consensus.cc:1050] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6: attempting to promote NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 to VOTER
I20240503 13:59:56.396150 14167 consensus_queue.cc:237] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 13:59:56.399452 9079 consensus_queue.cc:237] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 13:59:56.399516 6920 raft_consensus.cc:2932] T 61dfb1d40ee444368470a0dafc3179a4 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:56.399716 8947 raft_consensus.cc:1261] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:56.399544 6922 raft_consensus.cc:1261] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:56.400420 13080 consensus_queue.cc:1035] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:56.400579 11167 consensus_queue.cc:1035] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:56.401276 6526 raft_consensus.cc:1261] T 44eddd53c3f348a9a1117a20473cbb65 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.401662 12372 consensus_queue.cc:1035] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.402487 9342 raft_consensus.cc:1261] T 44eddd53c3f348a9a1117a20473cbb65 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.402842 12372 consensus_queue.cc:1035] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.404214 12864 raft_consensus.cc:2932] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:56.404760 6526 raft_consensus.cc:2932] T 44eddd53c3f348a9a1117a20473cbb65 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:56.406123 9342 raft_consensus.cc:2932] T 44eddd53c3f348a9a1117a20473cbb65 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:56.407212 6381 catalog_manager.cc:5484] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:56.407402 6659 consensus_queue.cc:237] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 13:59:56.407858 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 44eddd53c3f348a9a1117a20473cbb65 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:56.408977 9343 raft_consensus.cc:1261] T 61dfb1d40ee444368470a0dafc3179a4 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.409145 13080 raft_consensus.cc:2932] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:56.409411 6920 raft_consensus.cc:1261] T 61dfb1d40ee444368470a0dafc3179a4 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.409595 13325 consensus_queue.cc:1035] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.409785 13575 consensus_queue.cc:1035] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.410353 6920 raft_consensus.cc:2932] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:56.411487 13575 raft_consensus.cc:2932] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:56.412671 6788 raft_consensus.cc:1261] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:56.413215 6374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 61dfb1d40ee444368470a0dafc3179a4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:56.413749 9192 tablet_service.cc:1514] Processing DeleteTablet for tablet 44eddd53c3f348a9a1117a20473cbb65 with delete_type TABLET_DATA_TOMBSTONED (TS ab1d21935d6049888eb0ceccc3c9d735 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57960
I20240503 13:59:56.413689 6922 raft_consensus.cc:2932] T 61dfb1d40ee444368470a0dafc3179a4 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:56.414243 8947 raft_consensus.cc:2932] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:56.414443 6384 catalog_manager.cc:5484] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:56.415098 14228 tablet_replica.cc:330] T 44eddd53c3f348a9a1117a20473cbb65 P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 13:59:56.415237 14228 raft_consensus.cc:2229] T 44eddd53c3f348a9a1117a20473cbb65 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:56.415418 14228 raft_consensus.cc:2258] T 44eddd53c3f348a9a1117a20473cbb65 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:56.416116 9343 raft_consensus.cc:2932] T 61dfb1d40ee444368470a0dafc3179a4 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:56.416563 14228 ts_tablet_manager.cc:1875] T 44eddd53c3f348a9a1117a20473cbb65 P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:56.417688 11167 consensus_queue.cc:1035] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:56.418735 14228 ts_tablet_manager.cc:1888] T 44eddd53c3f348a9a1117a20473cbb65 P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:56.418798 14228 log.cc:1196] T 44eddd53c3f348a9a1117a20473cbb65 P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/44eddd53c3f348a9a1117a20473cbb65
I20240503 13:59:56.419732 6387 catalog_manager.cc:5484] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:56.419821 6788 raft_consensus.cc:2932] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:56.420400 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet 44eddd53c3f348a9a1117a20473cbb65 (table rebalancer_extra_table_42 [id=2c584aa1fb5a4bc4a40f13eaf00c9ddf]) successfully deleted
I20240503 13:59:56.422034 9060 tablet_service.cc:1514] Processing DeleteTablet for tablet 61dfb1d40ee444368470a0dafc3179a4 with delete_type TABLET_DATA_TOMBSTONED (TS 8d574fb4a89d4eea947c859470d44dd4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:56.422230 14192 tablet_replica.cc:330] T 61dfb1d40ee444368470a0dafc3179a4 P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:56.422360 14192 raft_consensus.cc:2229] T 61dfb1d40ee444368470a0dafc3179a4 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:56.422480 14192 raft_consensus.cc:2258] T 61dfb1d40ee444368470a0dafc3179a4 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:56.423251 14192 ts_tablet_manager.cc:1875] T 61dfb1d40ee444368470a0dafc3179a4 P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:56.425444 14192 ts_tablet_manager.cc:1888] T 61dfb1d40ee444368470a0dafc3179a4 P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:56.425524 14192 log.cc:1196] T 61dfb1d40ee444368470a0dafc3179a4 P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/61dfb1d40ee444368470a0dafc3179a4
I20240503 13:59:56.427773 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet 61dfb1d40ee444368470a0dafc3179a4 (table rebalancer_extra_table_14 [id=b4dfb7bd2b7f40e9ad8e6e5c28e9ee31]) successfully deleted
I20240503 13:59:56.456200 6788 consensus_queue.cc:237] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
W20240503 13:59:56.457957 6726 consensus_peers.cc:485] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 -> Peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): Couldn't send request to peer f58065db70b64ef8b1dc499aa6ae4460. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5780f4a85b146e2bfbbf7d2feaca316. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:56.459619 9210 raft_consensus.cc:1261] T b5780f4a85b146e2bfbbf7d2feaca316 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:56.460232 9079 raft_consensus.cc:1261] T b5780f4a85b146e2bfbbf7d2feaca316 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:56.460227 13276 consensus_queue.cc:1035] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:56.460618 13276 consensus_queue.cc:1035] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:56.461946 14230 raft_consensus.cc:2932] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:56.463681 9210 raft_consensus.cc:2932] T b5780f4a85b146e2bfbbf7d2feaca316 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:56.466243 6381 catalog_manager.cc:5484] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:56.468742 9079 raft_consensus.cc:2932] T b5780f4a85b146e2bfbbf7d2feaca316 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:56.478209 6659 consensus_queue.cc:237] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
W20240503 13:59:56.479770 6595 consensus_peers.cc:485] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 -> Peer 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): Couldn't send request to peer 5f46e19644fd4f4b825c86ba6af768da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63d2b0a8c3734720ba7d72a36d344cba. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:56.480044 9079 raft_consensus.cc:1261] T 63d2b0a8c3734720ba7d72a36d344cba P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:56.480403 14335 consensus_queue.cc:1035] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:56.480587 9210 raft_consensus.cc:1261] T 63d2b0a8c3734720ba7d72a36d344cba P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:56.481029 14335 consensus_queue.cc:1035] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:56.482060 14335 raft_consensus.cc:2932] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:56.483034 9210 raft_consensus.cc:2932] T 63d2b0a8c3734720ba7d72a36d344cba P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:56.484774 9079 raft_consensus.cc:2932] T 63d2b0a8c3734720ba7d72a36d344cba P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:56.486071 6379 catalog_manager.cc:5484] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:56.496906 14384 raft_consensus.cc:1050] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545: attempting to promote NON_VOTER 0d77752181e944698e809e325e5937e6 to VOTER
I20240503 13:59:56.498277 14384 consensus_queue.cc:237] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 13:59:56.500022 6526 raft_consensus.cc:1261] T ef19b1e5f15e49448c1b2919a24090d3 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:56.500015 9343 raft_consensus.cc:1261] T ef19b1e5f15e49448c1b2919a24090d3 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:56.500428 14384 consensus_queue.cc:1035] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:56.500650 12367 consensus_queue.cc:1035] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:56.500772 6659 raft_consensus.cc:1261] T ef19b1e5f15e49448c1b2919a24090d3 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:56.501102 12367 consensus_queue.cc:1035] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:56.502907 12367 raft_consensus.cc:2932] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:56.503712 6659 raft_consensus.cc:2932] T ef19b1e5f15e49448c1b2919a24090d3 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:56.504606 9343 raft_consensus.cc:2932] T ef19b1e5f15e49448c1b2919a24090d3 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:56.505528 6526 raft_consensus.cc:2932] T ef19b1e5f15e49448c1b2919a24090d3 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:56.507138 9079 consensus_queue.cc:237] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:56.507138 6381 catalog_manager.cc:5484] T ef19b1e5f15e49448c1b2919a24090d3 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
W20240503 13:59:56.509272 9016 consensus_peers.cc:485] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 -> Peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): Couldn't send request to peer f58065db70b64ef8b1dc499aa6ae4460. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f16b677f47f54a13a7f374cce34713bd. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:56.509557 8947 raft_consensus.cc:1261] T f16b677f47f54a13a7f374cce34713bd P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:56.510039 6659 raft_consensus.cc:1261] T f16b677f47f54a13a7f374cce34713bd P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:56.510099 13280 consensus_queue.cc:1035] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:56.510476 13280 consensus_queue.cc:1035] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:56.512053 8946 consensus_queue.cc:237] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 13:59:56.513206 6526 raft_consensus.cc:1261] T ef19b1e5f15e49448c1b2919a24090d3 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.513315 14146 raft_consensus.cc:2932] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:56.513638 9343 raft_consensus.cc:1261] T ef19b1e5f15e49448c1b2919a24090d3 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.513862 14390 consensus_queue.cc:1035] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.514142 13268 consensus_queue.cc:1035] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.514228 8946 raft_consensus.cc:2932] T f16b677f47f54a13a7f374cce34713bd P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:56.515969 14386 raft_consensus.cc:2932] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:56.516355 6526 raft_consensus.cc:2932] T ef19b1e5f15e49448c1b2919a24090d3 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:56.517004 6384 catalog_manager.cc:5484] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:56.517474 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ef19b1e5f15e49448c1b2919a24090d3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:56.519194 6659 raft_consensus.cc:2932] T f16b677f47f54a13a7f374cce34713bd P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:56.519853 6379 catalog_manager.cc:5484] T ef19b1e5f15e49448c1b2919a24090d3 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index 3 to 4, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:56.524366 9343 raft_consensus.cc:2932] T ef19b1e5f15e49448c1b2919a24090d3 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:56.526782 14219 raft_consensus.cc:1050] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460: attempting to promote NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 to VOTER
I20240503 13:59:56.527206 14219 consensus_queue.cc:237] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 13:59:56.528455 6639 tablet_service.cc:1514] Processing DeleteTablet for tablet ef19b1e5f15e49448c1b2919a24090d3 with delete_type TABLET_DATA_TOMBSTONED (TS 2145d15ab2d74c0287839266f1411a43 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:56.529366 14400 tablet_replica.cc:330] T ef19b1e5f15e49448c1b2919a24090d3 P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:56.529502 14400 raft_consensus.cc:2229] T ef19b1e5f15e49448c1b2919a24090d3 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:56.529556 9079 raft_consensus.cc:1261] T d002f7a57be04915b7b0b3dcb985f550 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.529649 14400 raft_consensus.cc:2258] T ef19b1e5f15e49448c1b2919a24090d3 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:56.529625 6788 raft_consensus.cc:1261] T d002f7a57be04915b7b0b3dcb985f550 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.529881 9210 raft_consensus.cc:1261] T d002f7a57be04915b7b0b3dcb985f550 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.529901 12748 raft_consensus.cc:1050] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460: attempting to promote NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 to VOTER
I20240503 13:59:56.530086 13370 consensus_queue.cc:1035] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.530313 12748 consensus_queue.cc:237] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 13:59:56.532565 14400 ts_tablet_manager.cc:1875] T ef19b1e5f15e49448c1b2919a24090d3 P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:56.532943 9210 raft_consensus.cc:1261] T 218b25613baf48319576e433c2ac4534 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:56.532946 13159 consensus_queue.cc:1035] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.533061 8946 raft_consensus.cc:1261] T 218b25613baf48319576e433c2ac4534 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:56.533264 13159 consensus_queue.cc:1035] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.533542 13159 consensus_queue.cc:1035] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:56.534128 12748 consensus_queue.cc:1035] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:56.534745 6526 raft_consensus.cc:1261] T 218b25613baf48319576e433c2ac4534 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:56.535121 14400 ts_tablet_manager.cc:1888] T ef19b1e5f15e49448c1b2919a24090d3 P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:56.535248 14400 log.cc:1196] T ef19b1e5f15e49448c1b2919a24090d3 P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/ef19b1e5f15e49448c1b2919a24090d3
I20240503 13:59:56.535450 13370 consensus_queue.cc:1035] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:56.536037 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet ef19b1e5f15e49448c1b2919a24090d3 (table rebalancer_extra_table_60 [id=7a1d16a1055e477183d59c2aa87d96fb]) successfully deleted
I20240503 13:59:56.536226 13370 raft_consensus.cc:2932] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:56.537452 13159 raft_consensus.cc:2932] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:56.538235 9079 raft_consensus.cc:2932] T d002f7a57be04915b7b0b3dcb985f550 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:56.538954 6659 consensus_queue.cc:237] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:56.539644 6526 raft_consensus.cc:2932] T 218b25613baf48319576e433c2ac4534 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:56.540120 9210 raft_consensus.cc:2932] T d002f7a57be04915b7b0b3dcb985f550 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:56.540508 6791 raft_consensus.cc:1261] T 3612cd35d8fa4300a1132fe43a63e6c1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.540882 13325 consensus_queue.cc:1035] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.540936 6384 catalog_manager.cc:5484] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 3 to 4, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:56.542378 6922 raft_consensus.cc:1261] T 3612cd35d8fa4300a1132fe43a63e6c1 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.543498 9210 raft_consensus.cc:2932] T 218b25613baf48319576e433c2ac4534 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:56.545046 6379 catalog_manager.cc:5484] T 218b25613baf48319576e433c2ac4534 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index 2 to 3, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
W20240503 13:59:56.545320 6596 consensus_peers.cc:485] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3612cd35d8fa4300a1132fe43a63e6c1. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:56.545882 13325 consensus_queue.cc:1035] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.548389 8946 raft_consensus.cc:2932] T 218b25613baf48319576e433c2ac4534 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:56.549791 13325 raft_consensus.cc:2932] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:56.550837 6788 raft_consensus.cc:2932] T d002f7a57be04915b7b0b3dcb985f550 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:56.552172 6920 consensus_queue.cc:237] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 13:59:56.552546 6922 raft_consensus.cc:2932] T 3612cd35d8fa4300a1132fe43a63e6c1 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:56.554324 9210 raft_consensus.cc:1261] T 218b25613baf48319576e433c2ac4534 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.554772 12748 consensus_queue.cc:1035] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.556720 13159 raft_consensus.cc:2932] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:56.557076 9210 raft_consensus.cc:2932] T 218b25613baf48319576e433c2ac4534 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:56.557267 6526 raft_consensus.cc:1261] T 218b25613baf48319576e433c2ac4534 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.557628 14219 consensus_queue.cc:1035] T 218b25613baf48319576e433c2ac4534 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.558163 6922 consensus_queue.cc:237] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 13:59:56.558789 6526 raft_consensus.cc:2932] T 218b25613baf48319576e433c2ac4534 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:56.559055 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 218b25613baf48319576e433c2ac4534 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:56.559803 12508 raft_consensus.cc:1050] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da: attempting to promote NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 to VOTER
I20240503 13:59:56.560074 6788 raft_consensus.cc:1261] T d002f7a57be04915b7b0b3dcb985f550 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:56.560154 12508 consensus_queue.cc:237] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 13:59:56.560484 14219 consensus_queue.cc:1035] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:56.560740 9079 raft_consensus.cc:1261] T d002f7a57be04915b7b0b3dcb985f550 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:56.561275 6379 catalog_manager.cc:5484] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:56.561512 6659 raft_consensus.cc:1261] T da69140936424a31b66c87e2ad4092cb P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:56.561686 6922 raft_consensus.cc:1261] T da69140936424a31b66c87e2ad4092cb P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:56.561892 13157 consensus_queue.cc:1035] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:56.562197 13157 consensus_queue.cc:1035] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:56.562599 14219 consensus_queue.cc:1035] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:56.563751 9210 raft_consensus.cc:1261] T da69140936424a31b66c87e2ad4092cb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:56.564298 14214 consensus_queue.cc:1035] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:56.564380 6384 catalog_manager.cc:5484] T 218b25613baf48319576e433c2ac4534 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:56.565411 14219 raft_consensus.cc:2932] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:56.567070 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d002f7a57be04915b7b0b3dcb985f550 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:56.567255 6788 raft_consensus.cc:2932] T d002f7a57be04915b7b0b3dcb985f550 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:56.569463 8926 tablet_service.cc:1514] Processing DeleteTablet for tablet 218b25613baf48319576e433c2ac4534 with delete_type TABLET_DATA_TOMBSTONED (TS 06ca8dbc004c42b99faa527a4e6f9545 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:56.569694 14181 tablet_replica.cc:330] T 218b25613baf48319576e433c2ac4534 P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:56.569844 14181 raft_consensus.cc:2229] T 218b25613baf48319576e433c2ac4534 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:56.569974 14181 raft_consensus.cc:2258] T 218b25613baf48319576e433c2ac4534 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:56.570863 6381 catalog_manager.cc:5484] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 4 to 5, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:56.572443 14181 ts_tablet_manager.cc:1875] T 218b25613baf48319576e433c2ac4534 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:56.574751 14181 ts_tablet_manager.cc:1888] T 218b25613baf48319576e433c2ac4534 P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:56.574827 14181 log.cc:1196] T 218b25613baf48319576e433c2ac4534 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/218b25613baf48319576e433c2ac4534
I20240503 13:59:56.574826 13157 raft_consensus.cc:2932] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:56.575575 6659 raft_consensus.cc:2932] T da69140936424a31b66c87e2ad4092cb P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:56.577270 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 218b25613baf48319576e433c2ac4534 (table rebalancer_extra_table_52 [id=17d62e19cb344d018bc5ae78ffca2185]) successfully deleted
I20240503 13:59:56.577162 6791 raft_consensus.cc:2932] T 3612cd35d8fa4300a1132fe43a63e6c1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:56.578310 6922 raft_consensus.cc:2932] T da69140936424a31b66c87e2ad4092cb P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:56.579728 9210 raft_consensus.cc:2932] T da69140936424a31b66c87e2ad4092cb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:56.582804 9079 raft_consensus.cc:2932] T d002f7a57be04915b7b0b3dcb985f550 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:56.583027 6381 catalog_manager.cc:5484] T da69140936424a31b66c87e2ad4092cb P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:56.583220 9192 tablet_service.cc:1514] Processing DeleteTablet for tablet d002f7a57be04915b7b0b3dcb985f550 with delete_type TABLET_DATA_TOMBSTONED (TS ab1d21935d6049888eb0ceccc3c9d735 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57960
I20240503 13:59:56.583415 14228 tablet_replica.cc:330] T d002f7a57be04915b7b0b3dcb985f550 P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 13:59:56.583536 14228 raft_consensus.cc:2229] T d002f7a57be04915b7b0b3dcb985f550 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:56.583659 14228 raft_consensus.cc:2258] T d002f7a57be04915b7b0b3dcb985f550 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:56.586370 14228 ts_tablet_manager.cc:1875] T d002f7a57be04915b7b0b3dcb985f550 P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:56.588335 14228 ts_tablet_manager.cc:1888] T d002f7a57be04915b7b0b3dcb985f550 P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 13:59:56.588423 14228 log.cc:1196] T d002f7a57be04915b7b0b3dcb985f550 P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/d002f7a57be04915b7b0b3dcb985f550
I20240503 13:59:56.592010 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet d002f7a57be04915b7b0b3dcb985f550 (table rebalancer_dynamic_table [id=c0369d036158436b8526abbc1683d235]) successfully deleted
I20240503 13:59:56.592756 6526 consensus_queue.cc:237] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 13:59:56.594419 9210 raft_consensus.cc:1261] T da69140936424a31b66c87e2ad4092cb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:56.594496 6920 raft_consensus.cc:1261] T da69140936424a31b66c87e2ad4092cb P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:56.594913 12508 consensus_queue.cc:1035] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:56.595089 13902 consensus_queue.cc:1035] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:56.595119 9081 consensus_queue.cc:237] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
W20240503 13:59:56.596765 9016 consensus_peers.cc:485] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): Couldn't send request to peer 5f46e19644fd4f4b825c86ba6af768da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 387d78763d574e57b914e2dbac83a932. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:56.596817 9211 raft_consensus.cc:1261] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:56.596943 13157 raft_consensus.cc:2932] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:56.597082 6791 raft_consensus.cc:1261] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:56.597265 12864 consensus_queue.cc:1035] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:56.597456 12372 consensus_queue.cc:1035] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:56.598789 6920 raft_consensus.cc:2932] T da69140936424a31b66c87e2ad4092cb P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:56.599398 12372 raft_consensus.cc:2932] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:56.599805 9210 raft_consensus.cc:2932] T da69140936424a31b66c87e2ad4092cb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:56.600319 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet da69140936424a31b66c87e2ad4092cb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:56.601486 9210 raft_consensus.cc:2932] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:56.603552 6384 catalog_manager.cc:5484] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:56.603749 6791 raft_consensus.cc:2932] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } } }
I20240503 13:59:56.605037 6385 catalog_manager.cc:5484] T da69140936424a31b66c87e2ad4092cb P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:56.609201 6639 tablet_service.cc:1514] Processing DeleteTablet for tablet da69140936424a31b66c87e2ad4092cb with delete_type TABLET_DATA_TOMBSTONED (TS 2145d15ab2d74c0287839266f1411a43 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:56.609448 14400 tablet_replica.cc:330] T da69140936424a31b66c87e2ad4092cb P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:56.609584 14400 raft_consensus.cc:2229] T da69140936424a31b66c87e2ad4092cb P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:56.609753 14400 raft_consensus.cc:2258] T da69140936424a31b66c87e2ad4092cb P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:56.610522 14400 ts_tablet_manager.cc:1875] T da69140936424a31b66c87e2ad4092cb P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:56.612706 14400 ts_tablet_manager.cc:1888] T da69140936424a31b66c87e2ad4092cb P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:56.612787 14400 log.cc:1196] T da69140936424a31b66c87e2ad4092cb P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/da69140936424a31b66c87e2ad4092cb
I20240503 13:59:56.613564 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet da69140936424a31b66c87e2ad4092cb (table rebalance_test_table_13 [id=37a0688b297f4755a4cc7c147a101e22]) successfully deleted
I20240503 13:59:56.617040 9343 consensus_queue.cc:237] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:56.618491 9210 raft_consensus.cc:1261] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:56.618666 9278 consensus_peers.cc:485] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 -> Peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): Couldn't send request to peer f58065db70b64ef8b1dc499aa6ae4460. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1e5875904e6047eb82f28c3cbc9318fc. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:56.618832 8946 raft_consensus.cc:1261] T 1e5875904e6047eb82f28c3cbc9318fc P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:56.619227 14167 consensus_queue.cc:1035] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:56.619580 14167 consensus_queue.cc:1035] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:56.621845 11167 raft_consensus.cc:2932] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:56.622110 9211 raft_consensus.cc:2932] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:56.623724 8946 raft_consensus.cc:2932] T 1e5875904e6047eb82f28c3cbc9318fc P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:56.625736 6380 catalog_manager.cc:5484] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:56.634792 6920 consensus_queue.cc:237] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:56.636584 6526 raft_consensus.cc:1261] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:59:56.636706 6857 consensus_peers.cc:485] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 -> Peer ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): Couldn't send request to peer ab1d21935d6049888eb0ceccc3c9d735. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c46822ccc1e48469c8d7f31b0bb0b8a. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:56.636792 6657 raft_consensus.cc:1261] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.637140 12748 consensus_queue.cc:1035] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.637499 12748 consensus_queue.cc:1035] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.637720 13116 raft_consensus.cc:1050] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64: attempting to promote NON_VOTER 0d77752181e944698e809e325e5937e6 to VOTER
I20240503 13:59:56.638123 13116 consensus_queue.cc:237] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 13:59:56.639364 14219 raft_consensus.cc:2932] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:56.639838 9343 raft_consensus.cc:1261] T ea45c42d7fed4171a1b726b0684d2fa5 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:56.639890 6920 raft_consensus.cc:1261] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:56.639890 6657 raft_consensus.cc:1261] T ea45c42d7fed4171a1b726b0684d2fa5 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:56.640291 13276 consensus_queue.cc:1035] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:56.640460 14230 consensus_queue.cc:1035] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:56.640762 13276 consensus_queue.cc:1035] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:56.643699 6657 raft_consensus.cc:2932] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:56.644176 6387 catalog_manager.cc:5484] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index -1 to 4, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:56.645334 6526 raft_consensus.cc:2932] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:56.653224 12032 raft_consensus.cc:2932] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:56.655779 6657 raft_consensus.cc:2932] T ea45c42d7fed4171a1b726b0684d2fa5 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:56.656013 9343 raft_consensus.cc:2932] T ea45c42d7fed4171a1b726b0684d2fa5 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:56.658195 6380 catalog_manager.cc:5484] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:56.661378 6920 raft_consensus.cc:2932] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:56.665910 6791 consensus_queue.cc:237] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 13:59:56.667943 6920 raft_consensus.cc:1261] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:56.668505 12032 consensus_queue.cc:1035] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:56.669127 9343 raft_consensus.cc:1261] T ea45c42d7fed4171a1b726b0684d2fa5 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:56.669481 12032 consensus_queue.cc:1035] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:56.670249 12032 raft_consensus.cc:2932] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:56.670540 6920 raft_consensus.cc:2932] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:56.671835 6372 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ea45c42d7fed4171a1b726b0684d2fa5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:56.672106 9343 raft_consensus.cc:2932] T ea45c42d7fed4171a1b726b0684d2fa5 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:56.672365 8946 consensus_queue.cc:237] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:56.674130 9081 raft_consensus.cc:1261] T fd45a0cf3a2d4657b02016bc60c5be34 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:56.674211 6387 catalog_manager.cc:5484] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:56.674630 13268 consensus_queue.cc:1035] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:56.674702 9342 raft_consensus.cc:1261] T fd45a0cf3a2d4657b02016bc60c5be34 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:56.675117 8882 consensus_peers.cc:485] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd45a0cf3a2d4657b02016bc60c5be34. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:56.675362 13268 consensus_queue.cc:1035] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:56.677021 6639 tablet_service.cc:1514] Processing DeleteTablet for tablet ea45c42d7fed4171a1b726b0684d2fa5 with delete_type TABLET_DATA_TOMBSTONED (TS 2145d15ab2d74c0287839266f1411a43 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:56.677279 14400 tablet_replica.cc:330] T ea45c42d7fed4171a1b726b0684d2fa5 P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:56.677402 14400 raft_consensus.cc:2229] T ea45c42d7fed4171a1b726b0684d2fa5 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:56.677515 14400 raft_consensus.cc:2258] T ea45c42d7fed4171a1b726b0684d2fa5 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:56.677395 13268 raft_consensus.cc:2932] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:56.678156 14400 ts_tablet_manager.cc:1875] T ea45c42d7fed4171a1b726b0684d2fa5 P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:56.678725 9342 raft_consensus.cc:2932] T fd45a0cf3a2d4657b02016bc60c5be34 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:56.680159 14400 ts_tablet_manager.cc:1888] T ea45c42d7fed4171a1b726b0684d2fa5 P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:56.680217 14400 log.cc:1196] T ea45c42d7fed4171a1b726b0684d2fa5 P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/ea45c42d7fed4171a1b726b0684d2fa5
I20240503 13:59:56.680583 9081 raft_consensus.cc:2932] T fd45a0cf3a2d4657b02016bc60c5be34 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:56.680783 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet ea45c42d7fed4171a1b726b0684d2fa5 (table rebalance_test_table_5 [id=cb716c6127694172a7a76a19a23dc99e]) successfully deleted
I20240503 13:59:56.682018 6379 catalog_manager.cc:5484] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:56.694638 8946 consensus_queue.cc:237] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
W20240503 13:59:56.695973 8883 consensus_peers.cc:485] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): Couldn't send request to peer f58065db70b64ef8b1dc499aa6ae4460. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 038537d00e904bca89f0214d04fb4f13. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:56.695958 9211 raft_consensus.cc:1261] T 038537d00e904bca89f0214d04fb4f13 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:56.696310 9081 raft_consensus.cc:1261] T 038537d00e904bca89f0214d04fb4f13 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:56.696342 14384 consensus_queue.cc:1035] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:56.696617 14386 consensus_queue.cc:1035] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:56.698089 14390 raft_consensus.cc:2932] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:56.698491 9081 raft_consensus.cc:2932] T 038537d00e904bca89f0214d04fb4f13 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:56.698535 9211 raft_consensus.cc:2932] T 038537d00e904bca89f0214d04fb4f13 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:56.701607 6379 catalog_manager.cc:5484] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:56.818257 6657 raft_consensus.cc:1203] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:56.829596 6791 consensus_queue.cc:237] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:56.831104 6657 raft_consensus.cc:1261] T ce9410a4ed09491b8d37456ae44880a1 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:59:56.831226 6726 consensus_peers.cc:485] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 -> Peer ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): Couldn't send request to peer ab1d21935d6049888eb0ceccc3c9d735. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce9410a4ed09491b8d37456ae44880a1. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:56.831566 6526 raft_consensus.cc:1261] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.831796 13276 consensus_queue.cc:1035] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.831966 12032 consensus_queue.cc:1035] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.833788 13276 raft_consensus.cc:2932] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:56.835367 6526 raft_consensus.cc:2932] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:56.835817 6657 raft_consensus.cc:2932] T ce9410a4ed09491b8d37456ae44880a1 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:56.840618 6379 catalog_manager.cc:5484] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index -1 to 4, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:56.849669 9081 consensus_queue.cc:237] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 13:59:56.851377 9211 raft_consensus.cc:1261] T b8c8bc1c29934e32a3f30ad20db05462 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:56.851459 6920 raft_consensus.cc:1261] T b8c8bc1c29934e32a3f30ad20db05462 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:56.851698 9016 consensus_peers.cc:485] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): Couldn't send request to peer 06ca8dbc004c42b99faa527a4e6f9545. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8c8bc1c29934e32a3f30ad20db05462. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:56.851912 12372 consensus_queue.cc:1035] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:56.852077 12864 consensus_queue.cc:1035] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:56.853931 14146 raft_consensus.cc:2932] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:56.854260 6920 raft_consensus.cc:2932] T b8c8bc1c29934e32a3f30ad20db05462 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:56.855033 9211 raft_consensus.cc:2932] T b8c8bc1c29934e32a3f30ad20db05462 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:56.858108 6380 catalog_manager.cc:5484] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index -1 to 2, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:56.867085 6920 consensus_queue.cc:237] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:56.868496 6657 raft_consensus.cc:1261] T af98fddcb623403c841a77ecc10a3e07 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.868567 6791 raft_consensus.cc:1261] T af98fddcb623403c841a77ecc10a3e07 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.868904 14408 consensus_queue.cc:1035] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.869060 12748 consensus_queue.cc:1035] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:59:56.869187 6858 consensus_peers.cc:485] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af98fddcb623403c841a77ecc10a3e07. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:56.870731 13370 raft_consensus.cc:2932] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:56.871186 6791 raft_consensus.cc:2932] T af98fddcb623403c841a77ecc10a3e07 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:56.871323 6657 raft_consensus.cc:2932] T af98fddcb623403c841a77ecc10a3e07 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:56.874451 6387 catalog_manager.cc:5484] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:56.878321 14581 ts_tablet_manager.cc:902] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460: Initiating tablet copy from peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:56.878810 14581 tablet_copy_client.cc:308] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:46365
I20240503 13:59:56.879326 6811 tablet_copy_service.cc:140] P 4a85e492c8154e9881a9cc2a105d7e64: Received BeginTabletCopySession request for tablet b5780f4a85b146e2bfbbf7d2feaca316 from peer f58065db70b64ef8b1dc499aa6ae4460 ({username='slave'} at 127.0.99.132:42563)
I20240503 13:59:56.879419 6811 tablet_copy_service.cc:161] P 4a85e492c8154e9881a9cc2a105d7e64: Beginning new tablet copy session on tablet b5780f4a85b146e2bfbbf7d2feaca316 from peer f58065db70b64ef8b1dc499aa6ae4460 at {username='slave'} at 127.0.99.132:42563: session id = f58065db70b64ef8b1dc499aa6ae4460-b5780f4a85b146e2bfbbf7d2feaca316
I20240503 13:59:56.880378 6811 tablet_copy_source_session.cc:205] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:56.881362 9342 consensus_queue.cc:237] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 13:59:56.881585 14581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5780f4a85b146e2bfbbf7d2feaca316. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:56.883023 6791 raft_consensus.cc:1261] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.883352 6920 raft_consensus.cc:1261] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.883672 13080 consensus_queue.cc:1035] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.883988 13080 consensus_queue.cc:1035] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.884475 14581 tablet_copy_client.cc:783] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:56.884627 14581 tablet_copy_client.cc:647] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:56.886245 13080 raft_consensus.cc:2932] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:56.887212 6920 raft_consensus.cc:2932] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:56.887495 6791 raft_consensus.cc:2932] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:56.890558 6374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7a25af1e7ae64f4fad8ad82faa1c6eb6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:56.891196 14581 tablet_copy_client.cc:523] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:56.892593 14581 tablet_bootstrap.cc:492] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:56.891669 6387 catalog_manager.cc:5484] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:56.894148 8946 consensus_queue.cc:237] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:56.895380 8926 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a25af1e7ae64f4fad8ad82faa1c6eb6 with delete_type TABLET_DATA_TOMBSTONED (TS 06ca8dbc004c42b99faa527a4e6f9545 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:56.895586 14181 tablet_replica.cc:330] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:56.895732 14181 raft_consensus.cc:2229] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:56.895862 14181 raft_consensus.cc:2258] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:56.896432 14181 ts_tablet_manager.cc:1875] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:56.897595 9211 raft_consensus.cc:1261] T 174e5b7577d94d87b729d1735987a97a P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:56.897961 9342 raft_consensus.cc:1261] T 174e5b7577d94d87b729d1735987a97a P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:56.898460 13268 consensus_queue.cc:1035] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:56.898597 14181 ts_tablet_manager.cc:1888] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:56.898660 14181 log.cc:1196] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/7a25af1e7ae64f4fad8ad82faa1c6eb6
I20240503 13:59:56.898797 13268 consensus_queue.cc:1035] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:56.899387 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 7a25af1e7ae64f4fad8ad82faa1c6eb6 (table rebalancer_extra_table_34 [id=22ad55d53adb4db5b7dd551f6384d11b]) successfully deleted
W20240503 13:59:56.899827 8883 consensus_peers.cc:485] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): Couldn't send request to peer 8d574fb4a89d4eea947c859470d44dd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 174e5b7577d94d87b729d1735987a97a. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:56.901095 14384 raft_consensus.cc:2932] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:56.902552 14581 tablet_bootstrap.cc:492] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:56.902832 14581 tablet_bootstrap.cc:492] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap complete.
I20240503 13:59:56.902988 14581 ts_tablet_manager.cc:1367] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.001s
I20240503 13:59:56.903453 14581 raft_consensus.cc:348] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:56.903569 14581 raft_consensus.cc:729] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: LEARNER
I20240503 13:59:56.903506 9343 raft_consensus.cc:2932] T 174e5b7577d94d87b729d1735987a97a P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:56.903663 14581 consensus_queue.cc:260] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:56.903987 14581 ts_tablet_manager.cc:1398] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:56.904170 6379 catalog_manager.cc:5484] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index -1 to 2, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:56.905479 6811 tablet_copy_service.cc:342] P 4a85e492c8154e9881a9cc2a105d7e64: Request end of tablet copy session f58065db70b64ef8b1dc499aa6ae4460-b5780f4a85b146e2bfbbf7d2feaca316 received from {username='slave'} at 127.0.99.132:42563
I20240503 13:59:56.905562 6811 tablet_copy_service.cc:433] P 4a85e492c8154e9881a9cc2a105d7e64: ending tablet copy session f58065db70b64ef8b1dc499aa6ae4460-b5780f4a85b146e2bfbbf7d2feaca316 on tablet b5780f4a85b146e2bfbbf7d2feaca316 with peer f58065db70b64ef8b1dc499aa6ae4460
I20240503 13:59:56.909224 6791 raft_consensus.cc:1203] T 9f0daf41253f4678b158e3ccbb7b0c98 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:56.912817 9211 raft_consensus.cc:2932] T 174e5b7577d94d87b729d1735987a97a P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:56.920096 6657 consensus_queue.cc:237] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 13:59:56.921558 6920 raft_consensus.cc:1261] T efe57c7f306b4b0ab934388f5a372a74 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.921569 6791 raft_consensus.cc:1261] T efe57c7f306b4b0ab934388f5a372a74 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.921936 13575 consensus_queue.cc:1035] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:59:56.922297 6595 consensus_peers.cc:485] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 -> Peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): Couldn't send request to peer 06ca8dbc004c42b99faa527a4e6f9545. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: efe57c7f306b4b0ab934388f5a372a74. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:56.922506 13575 consensus_queue.cc:1035] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.923658 12915 raft_consensus.cc:2932] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:56.923985 6791 raft_consensus.cc:2932] T efe57c7f306b4b0ab934388f5a372a74 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:56.925865 6920 raft_consensus.cc:2932] T efe57c7f306b4b0ab934388f5a372a74 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:56.927402 6380 catalog_manager.cc:5484] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:56.940568 6920 consensus_queue.cc:237] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
W20240503 13:59:56.942268 6857 consensus_peers.cc:485] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): Couldn't send request to peer ab1d21935d6049888eb0ceccc3c9d735. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3331108688a3485a9c44c65600f0f3e5. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:56.942704 6791 raft_consensus.cc:1261] T 3331108688a3485a9c44c65600f0f3e5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.943159 6526 raft_consensus.cc:1261] T 3331108688a3485a9c44c65600f0f3e5 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:56.943324 13370 consensus_queue.cc:1035] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.943794 13370 consensus_queue.cc:1035] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:56.944980 14219 raft_consensus.cc:2932] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:56.945397 6791 raft_consensus.cc:2932] T 3331108688a3485a9c44c65600f0f3e5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:56.946972 14603 ts_tablet_manager.cc:902] T 3612cd35d8fa4300a1132fe43a63e6c1 P 0d77752181e944698e809e325e5937e6: Initiating tablet copy from peer 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:56.948285 6379 catalog_manager.cc:5484] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index -1 to 4, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:56.951697 14603 tablet_copy_client.cc:308] T 3612cd35d8fa4300a1132fe43a63e6c1 P 0d77752181e944698e809e325e5937e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46119
I20240503 13:59:56.952118 6680 tablet_copy_service.cc:140] P 2145d15ab2d74c0287839266f1411a43: Received BeginTabletCopySession request for tablet 3612cd35d8fa4300a1132fe43a63e6c1 from peer 0d77752181e944698e809e325e5937e6 ({username='slave'} at 127.0.99.136:46439)
I20240503 13:59:56.952221 6680 tablet_copy_service.cc:161] P 2145d15ab2d74c0287839266f1411a43: Beginning new tablet copy session on tablet 3612cd35d8fa4300a1132fe43a63e6c1 from peer 0d77752181e944698e809e325e5937e6 at {username='slave'} at 127.0.99.136:46439: session id = 0d77752181e944698e809e325e5937e6-3612cd35d8fa4300a1132fe43a63e6c1
I20240503 13:59:56.953114 6680 tablet_copy_source_session.cc:205] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:56.953949 14603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3612cd35d8fa4300a1132fe43a63e6c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:56.956032 14603 tablet_copy_client.cc:783] T 3612cd35d8fa4300a1132fe43a63e6c1 P 0d77752181e944698e809e325e5937e6: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:56.956202 14603 tablet_copy_client.cc:647] T 3612cd35d8fa4300a1132fe43a63e6c1 P 0d77752181e944698e809e325e5937e6: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:56.957535 14603 tablet_copy_client.cc:523] T 3612cd35d8fa4300a1132fe43a63e6c1 P 0d77752181e944698e809e325e5937e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:56.957929 6920 consensus_queue.cc:237] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:56.958855 14603 tablet_bootstrap.cc:492] T 3612cd35d8fa4300a1132fe43a63e6c1 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:56.959771 6528 raft_consensus.cc:1261] T f403dc06195d449e8f454a9f4e795925 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:56.960119 6657 raft_consensus.cc:1261] T f403dc06195d449e8f454a9f4e795925 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:56.960422 13370 consensus_queue.cc:1035] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:56.962556 14408 consensus_queue.cc:1035] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 13:59:56.962445 6858 consensus_peers.cc:485] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f403dc06195d449e8f454a9f4e795925. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:56.964450 6528 raft_consensus.cc:2932] T 3331108688a3485a9c44c65600f0f3e5 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:56.968601 14408 raft_consensus.cc:2932] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:56.970547 6657 raft_consensus.cc:2932] T f403dc06195d449e8f454a9f4e795925 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:56.972085 6526 raft_consensus.cc:2932] T f403dc06195d449e8f454a9f4e795925 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:56.974769 6380 catalog_manager.cc:5484] T f403dc06195d449e8f454a9f4e795925 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index -1 to 2, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:56.995102 14603 tablet_bootstrap.cc:492] T 3612cd35d8fa4300a1132fe43a63e6c1 P 0d77752181e944698e809e325e5937e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:56.995558 14603 tablet_bootstrap.cc:492] T 3612cd35d8fa4300a1132fe43a63e6c1 P 0d77752181e944698e809e325e5937e6: Bootstrap complete.
I20240503 13:59:56.995751 14603 ts_tablet_manager.cc:1367] T 3612cd35d8fa4300a1132fe43a63e6c1 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.037s user 0.018s sys 0.008s
I20240503 13:59:56.996459 14581 ts_tablet_manager.cc:902] T f16b677f47f54a13a7f374cce34713bd P f58065db70b64ef8b1dc499aa6ae4460: Initiating tablet copy from peer 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:56.996930 14581 tablet_copy_client.cc:308] T f16b677f47f54a13a7f374cce34713bd P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:39481
I20240503 13:59:56.997349 9101 tablet_copy_service.cc:140] P 8d574fb4a89d4eea947c859470d44dd4: Received BeginTabletCopySession request for tablet f16b677f47f54a13a7f374cce34713bd from peer f58065db70b64ef8b1dc499aa6ae4460 ({username='slave'} at 127.0.99.132:53261)
I20240503 13:59:56.997452 9101 tablet_copy_service.cc:161] P 8d574fb4a89d4eea947c859470d44dd4: Beginning new tablet copy session on tablet f16b677f47f54a13a7f374cce34713bd from peer f58065db70b64ef8b1dc499aa6ae4460 at {username='slave'} at 127.0.99.132:53261: session id = f58065db70b64ef8b1dc499aa6ae4460-f16b677f47f54a13a7f374cce34713bd
I20240503 13:59:56.998378 9101 tablet_copy_source_session.cc:205] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:56.996276 14603 raft_consensus.cc:348] T 3612cd35d8fa4300a1132fe43a63e6c1 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:56.998874 14603 raft_consensus.cc:729] T 3612cd35d8fa4300a1132fe43a63e6c1 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: LEARNER
I20240503 13:59:56.999326 14581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f16b677f47f54a13a7f374cce34713bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:56.999032 14603 consensus_queue.cc:260] T 3612cd35d8fa4300a1132fe43a63e6c1 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:57.000047 14644 ts_tablet_manager.cc:902] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460: Initiating tablet copy from peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:57.000700 14644 tablet_copy_client.cc:308] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:41931
I20240503 13:59:57.001101 9362 tablet_copy_service.cc:140] P 0d77752181e944698e809e325e5937e6: Received BeginTabletCopySession request for tablet 1e5875904e6047eb82f28c3cbc9318fc from peer f58065db70b64ef8b1dc499aa6ae4460 ({username='slave'} at 127.0.99.132:60419)
I20240503 13:59:57.001185 9362 tablet_copy_service.cc:161] P 0d77752181e944698e809e325e5937e6: Beginning new tablet copy session on tablet 1e5875904e6047eb82f28c3cbc9318fc from peer f58065db70b64ef8b1dc499aa6ae4460 at {username='slave'} at 127.0.99.132:60419: session id = f58065db70b64ef8b1dc499aa6ae4460-1e5875904e6047eb82f28c3cbc9318fc
I20240503 13:59:57.001987 9362 tablet_copy_source_session.cc:205] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:57.002470 14603 ts_tablet_manager.cc:1398] T 3612cd35d8fa4300a1132fe43a63e6c1 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 13:59:57.002826 14644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e5875904e6047eb82f28c3cbc9318fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:57.004316 14581 tablet_copy_client.cc:783] T f16b677f47f54a13a7f374cce34713bd P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:57.004916 14581 tablet_copy_client.cc:647] T f16b677f47f54a13a7f374cce34713bd P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:57.005139 14644 tablet_copy_client.cc:783] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:57.005285 14644 tablet_copy_client.cc:647] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:57.006695 14581 tablet_copy_client.cc:523] T f16b677f47f54a13a7f374cce34713bd P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:57.006807 14644 tablet_copy_client.cc:523] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:57.007052 6680 tablet_copy_service.cc:342] P 2145d15ab2d74c0287839266f1411a43: Request end of tablet copy session 0d77752181e944698e809e325e5937e6-3612cd35d8fa4300a1132fe43a63e6c1 received from {username='slave'} at 127.0.99.136:46439
I20240503 13:59:57.007130 6680 tablet_copy_service.cc:433] P 2145d15ab2d74c0287839266f1411a43: ending tablet copy session 0d77752181e944698e809e325e5937e6-3612cd35d8fa4300a1132fe43a63e6c1 on tablet 3612cd35d8fa4300a1132fe43a63e6c1 with peer 0d77752181e944698e809e325e5937e6
I20240503 13:59:57.008476 14644 tablet_bootstrap.cc:492] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:57.008729 14581 tablet_bootstrap.cc:492] T f16b677f47f54a13a7f374cce34713bd P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:57.012692 14644 tablet_bootstrap.cc:492] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:57.012805 14581 tablet_bootstrap.cc:492] T f16b677f47f54a13a7f374cce34713bd P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:57.012946 14644 tablet_bootstrap.cc:492] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap complete.
I20240503 13:59:57.013044 14581 tablet_bootstrap.cc:492] T f16b677f47f54a13a7f374cce34713bd P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap complete.
I20240503 13:59:57.013089 14644 ts_tablet_manager.cc:1367] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 13:59:57.013185 14581 ts_tablet_manager.cc:1367] T f16b677f47f54a13a7f374cce34713bd P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 13:59:57.013589 14581 raft_consensus.cc:348] T f16b677f47f54a13a7f374cce34713bd P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:57.013705 14581 raft_consensus.cc:729] T f16b677f47f54a13a7f374cce34713bd P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: LEARNER
I20240503 13:59:57.013656 14644 raft_consensus.cc:348] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:57.013767 14644 raft_consensus.cc:729] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: LEARNER
I20240503 13:59:57.013803 14581 consensus_queue.cc:260] T f16b677f47f54a13a7f374cce34713bd P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:57.013860 14644 consensus_queue.cc:260] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:57.015340 14581 ts_tablet_manager.cc:1398] T f16b677f47f54a13a7f374cce34713bd P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:57.015482 14644 ts_tablet_manager.cc:1398] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:57.015822 9101 tablet_copy_service.cc:342] P 8d574fb4a89d4eea947c859470d44dd4: Request end of tablet copy session f58065db70b64ef8b1dc499aa6ae4460-f16b677f47f54a13a7f374cce34713bd received from {username='slave'} at 127.0.99.132:53261
I20240503 13:59:57.015895 9101 tablet_copy_service.cc:433] P 8d574fb4a89d4eea947c859470d44dd4: ending tablet copy session f58065db70b64ef8b1dc499aa6ae4460-f16b677f47f54a13a7f374cce34713bd on tablet f16b677f47f54a13a7f374cce34713bd with peer f58065db70b64ef8b1dc499aa6ae4460
I20240503 13:59:57.016062 9362 tablet_copy_service.cc:342] P 0d77752181e944698e809e325e5937e6: Request end of tablet copy session f58065db70b64ef8b1dc499aa6ae4460-1e5875904e6047eb82f28c3cbc9318fc received from {username='slave'} at 127.0.99.132:60419
I20240503 13:59:57.016132 9362 tablet_copy_service.cc:433] P 0d77752181e944698e809e325e5937e6: ending tablet copy session f58065db70b64ef8b1dc499aa6ae4460-1e5875904e6047eb82f28c3cbc9318fc on tablet 1e5875904e6047eb82f28c3cbc9318fc with peer f58065db70b64ef8b1dc499aa6ae4460
I20240503 13:59:57.031121 14666 ts_tablet_manager.cc:902] T 7c46822ccc1e48469c8d7f31b0bb0b8a P ab1d21935d6049888eb0ceccc3c9d735: Initiating tablet copy from peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:57.031704 14666 tablet_copy_client.cc:308] T 7c46822ccc1e48469c8d7f31b0bb0b8a P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43531
I20240503 13:59:57.032135 6942 tablet_copy_service.cc:140] P f58065db70b64ef8b1dc499aa6ae4460: Received BeginTabletCopySession request for tablet 7c46822ccc1e48469c8d7f31b0bb0b8a from peer ab1d21935d6049888eb0ceccc3c9d735 ({username='slave'} at 127.0.99.135:34893)
I20240503 13:59:57.032231 6942 tablet_copy_service.cc:161] P f58065db70b64ef8b1dc499aa6ae4460: Beginning new tablet copy session on tablet 7c46822ccc1e48469c8d7f31b0bb0b8a from peer ab1d21935d6049888eb0ceccc3c9d735 at {username='slave'} at 127.0.99.135:34893: session id = ab1d21935d6049888eb0ceccc3c9d735-7c46822ccc1e48469c8d7f31b0bb0b8a
I20240503 13:59:57.034124 6942 tablet_copy_source_session.cc:205] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:57.035032 14666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c46822ccc1e48469c8d7f31b0bb0b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:57.037354 14666 tablet_copy_client.cc:783] T 7c46822ccc1e48469c8d7f31b0bb0b8a P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:57.037496 14666 tablet_copy_client.cc:647] T 7c46822ccc1e48469c8d7f31b0bb0b8a P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:57.039669 14666 tablet_copy_client.cc:523] T 7c46822ccc1e48469c8d7f31b0bb0b8a P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:57.042130 14666 tablet_bootstrap.cc:492] T 7c46822ccc1e48469c8d7f31b0bb0b8a P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:57.068688 14666 tablet_bootstrap.cc:492] T 7c46822ccc1e48469c8d7f31b0bb0b8a P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:57.069077 14666 tablet_bootstrap.cc:492] T 7c46822ccc1e48469c8d7f31b0bb0b8a P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap complete.
I20240503 13:59:57.069268 14666 ts_tablet_manager.cc:1367] T 7c46822ccc1e48469c8d7f31b0bb0b8a P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.027s user 0.026s sys 0.000s
I20240503 13:59:57.069656 14666 raft_consensus.cc:348] T 7c46822ccc1e48469c8d7f31b0bb0b8a P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:57.069777 14666 raft_consensus.cc:729] T 7c46822ccc1e48469c8d7f31b0bb0b8a P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: LEARNER
I20240503 13:59:57.069881 14666 consensus_queue.cc:260] T 7c46822ccc1e48469c8d7f31b0bb0b8a P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:57.070241 14666 ts_tablet_manager.cc:1398] T 7c46822ccc1e48469c8d7f31b0bb0b8a P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:57.070791 6942 tablet_copy_service.cc:342] P f58065db70b64ef8b1dc499aa6ae4460: Request end of tablet copy session ab1d21935d6049888eb0ceccc3c9d735-7c46822ccc1e48469c8d7f31b0bb0b8a received from {username='slave'} at 127.0.99.135:34893
I20240503 13:59:57.070871 6942 tablet_copy_service.cc:433] P f58065db70b64ef8b1dc499aa6ae4460: ending tablet copy session ab1d21935d6049888eb0ceccc3c9d735-7c46822ccc1e48469c8d7f31b0bb0b8a on tablet 7c46822ccc1e48469c8d7f31b0bb0b8a with peer ab1d21935d6049888eb0ceccc3c9d735
I20240503 13:59:57.077585 14692 ts_tablet_manager.cc:902] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da: Initiating tablet copy from peer 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:57.079653 14692 tablet_copy_client.cc:308] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:39481
I20240503 13:59:57.080228 9101 tablet_copy_service.cc:140] P 8d574fb4a89d4eea947c859470d44dd4: Received BeginTabletCopySession request for tablet 387d78763d574e57b914e2dbac83a932 from peer 5f46e19644fd4f4b825c86ba6af768da ({username='slave'} at 127.0.99.129:55581)
I20240503 13:59:57.080348 9101 tablet_copy_service.cc:161] P 8d574fb4a89d4eea947c859470d44dd4: Beginning new tablet copy session on tablet 387d78763d574e57b914e2dbac83a932 from peer 5f46e19644fd4f4b825c86ba6af768da at {username='slave'} at 127.0.99.129:55581: session id = 5f46e19644fd4f4b825c86ba6af768da-387d78763d574e57b914e2dbac83a932
I20240503 13:59:57.081315 9101 tablet_copy_source_session.cc:205] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:57.082398 14692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387d78763d574e57b914e2dbac83a932. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:57.084439 14692 tablet_copy_client.cc:783] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:57.084654 14692 tablet_copy_client.cc:647] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:57.086433 14692 tablet_copy_client.cc:523] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:57.087723 14692 tablet_bootstrap.cc:492] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:57.093523 14692 tablet_bootstrap.cc:492] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:57.093869 14692 tablet_bootstrap.cc:492] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap complete.
I20240503 13:59:57.094070 14692 ts_tablet_manager.cc:1367] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240503 13:59:57.094521 14692 raft_consensus.cc:348] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:57.094667 14692 raft_consensus.cc:729] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: LEARNER
I20240503 13:59:57.094787 14692 consensus_queue.cc:260] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:57.095124 14692 ts_tablet_manager.cc:1398] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:57.095608 9101 tablet_copy_service.cc:342] P 8d574fb4a89d4eea947c859470d44dd4: Request end of tablet copy session 5f46e19644fd4f4b825c86ba6af768da-387d78763d574e57b914e2dbac83a932 received from {username='slave'} at 127.0.99.129:55581
I20240503 13:59:57.095705 9101 tablet_copy_service.cc:433] P 8d574fb4a89d4eea947c859470d44dd4: ending tablet copy session 5f46e19644fd4f4b825c86ba6af768da-387d78763d574e57b914e2dbac83a932 on tablet 387d78763d574e57b914e2dbac83a932 with peer 5f46e19644fd4f4b825c86ba6af768da
I20240503 13:59:57.098356 14692 ts_tablet_manager.cc:902] T 63d2b0a8c3734720ba7d72a36d344cba P 5f46e19644fd4f4b825c86ba6af768da: Initiating tablet copy from peer 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:57.098755 14692 tablet_copy_client.cc:308] T 63d2b0a8c3734720ba7d72a36d344cba P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46119
I20240503 13:59:57.099153 6680 tablet_copy_service.cc:140] P 2145d15ab2d74c0287839266f1411a43: Received BeginTabletCopySession request for tablet 63d2b0a8c3734720ba7d72a36d344cba from peer 5f46e19644fd4f4b825c86ba6af768da ({username='slave'} at 127.0.99.129:60979)
I20240503 13:59:57.099248 6680 tablet_copy_service.cc:161] P 2145d15ab2d74c0287839266f1411a43: Beginning new tablet copy session on tablet 63d2b0a8c3734720ba7d72a36d344cba from peer 5f46e19644fd4f4b825c86ba6af768da at {username='slave'} at 127.0.99.129:60979: session id = 5f46e19644fd4f4b825c86ba6af768da-63d2b0a8c3734720ba7d72a36d344cba
I20240503 13:59:57.100302 6680 tablet_copy_source_session.cc:205] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:57.101138 14692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63d2b0a8c3734720ba7d72a36d344cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:57.103199 14692 tablet_copy_client.cc:783] T 63d2b0a8c3734720ba7d72a36d344cba P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:57.103364 14692 tablet_copy_client.cc:647] T 63d2b0a8c3734720ba7d72a36d344cba P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:57.104405 14692 tablet_copy_client.cc:523] T 63d2b0a8c3734720ba7d72a36d344cba P 5f46e19644fd4f4b825c86ba6af768da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:57.108094 14692 tablet_bootstrap.cc:492] T 63d2b0a8c3734720ba7d72a36d344cba P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap starting.
I20240503 13:59:57.114006 14692 tablet_bootstrap.cc:492] T 63d2b0a8c3734720ba7d72a36d344cba P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:57.114248 14692 tablet_bootstrap.cc:492] T 63d2b0a8c3734720ba7d72a36d344cba P 5f46e19644fd4f4b825c86ba6af768da: Bootstrap complete.
I20240503 13:59:57.114424 14692 ts_tablet_manager.cc:1367] T 63d2b0a8c3734720ba7d72a36d344cba P 5f46e19644fd4f4b825c86ba6af768da: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20240503 13:59:57.114837 14692 raft_consensus.cc:348] T 63d2b0a8c3734720ba7d72a36d344cba P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:57.114974 14692 raft_consensus.cc:729] T 63d2b0a8c3734720ba7d72a36d344cba P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Initialized, Role: LEARNER
I20240503 13:59:57.115067 14692 consensus_queue.cc:260] T 63d2b0a8c3734720ba7d72a36d344cba P 5f46e19644fd4f4b825c86ba6af768da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: true } }
I20240503 13:59:57.115370 14692 ts_tablet_manager.cc:1398] T 63d2b0a8c3734720ba7d72a36d344cba P 5f46e19644fd4f4b825c86ba6af768da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:57.116459 6680 tablet_copy_service.cc:342] P 2145d15ab2d74c0287839266f1411a43: Request end of tablet copy session 5f46e19644fd4f4b825c86ba6af768da-63d2b0a8c3734720ba7d72a36d344cba received from {username='slave'} at 127.0.99.129:60979
I20240503 13:59:57.116537 6680 tablet_copy_service.cc:433] P 2145d15ab2d74c0287839266f1411a43: ending tablet copy session 5f46e19644fd4f4b825c86ba6af768da-63d2b0a8c3734720ba7d72a36d344cba on tablet 63d2b0a8c3734720ba7d72a36d344cba with peer 5f46e19644fd4f4b825c86ba6af768da
I20240503 13:59:57.123788 14729 ts_tablet_manager.cc:902] T fd45a0cf3a2d4657b02016bc60c5be34 P 4a85e492c8154e9881a9cc2a105d7e64: Initiating tablet copy from peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:57.124297 14729 tablet_copy_client.cc:308] T fd45a0cf3a2d4657b02016bc60c5be34 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:34547
I20240503 13:59:57.124867 8968 tablet_copy_service.cc:140] P 06ca8dbc004c42b99faa527a4e6f9545: Received BeginTabletCopySession request for tablet fd45a0cf3a2d4657b02016bc60c5be34 from peer 4a85e492c8154e9881a9cc2a105d7e64 ({username='slave'} at 127.0.99.131:54275)
I20240503 13:59:57.124969 8968 tablet_copy_service.cc:161] P 06ca8dbc004c42b99faa527a4e6f9545: Beginning new tablet copy session on tablet fd45a0cf3a2d4657b02016bc60c5be34 from peer 4a85e492c8154e9881a9cc2a105d7e64 at {username='slave'} at 127.0.99.131:54275: session id = 4a85e492c8154e9881a9cc2a105d7e64-fd45a0cf3a2d4657b02016bc60c5be34
I20240503 13:59:57.125839 8968 tablet_copy_source_session.cc:205] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:57.126891 14729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd45a0cf3a2d4657b02016bc60c5be34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:57.129312 14729 tablet_copy_client.cc:783] T fd45a0cf3a2d4657b02016bc60c5be34 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:57.129460 14729 tablet_copy_client.cc:647] T fd45a0cf3a2d4657b02016bc60c5be34 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:57.130576 14729 tablet_copy_client.cc:523] T fd45a0cf3a2d4657b02016bc60c5be34 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:57.131809 14729 tablet_bootstrap.cc:492] T fd45a0cf3a2d4657b02016bc60c5be34 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:57.135382 14729 tablet_bootstrap.cc:492] T fd45a0cf3a2d4657b02016bc60c5be34 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:57.135643 14729 tablet_bootstrap.cc:492] T fd45a0cf3a2d4657b02016bc60c5be34 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap complete.
I20240503 13:59:57.136366 14729 ts_tablet_manager.cc:1367] T fd45a0cf3a2d4657b02016bc60c5be34 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240503 13:59:57.136829 14729 raft_consensus.cc:348] T fd45a0cf3a2d4657b02016bc60c5be34 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:57.136936 14729 raft_consensus.cc:729] T fd45a0cf3a2d4657b02016bc60c5be34 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: LEARNER
I20240503 13:59:57.137022 14729 consensus_queue.cc:260] T fd45a0cf3a2d4657b02016bc60c5be34 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:57.137281 14729 ts_tablet_manager.cc:1398] T fd45a0cf3a2d4657b02016bc60c5be34 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:57.138567 8968 tablet_copy_service.cc:342] P 06ca8dbc004c42b99faa527a4e6f9545: Request end of tablet copy session 4a85e492c8154e9881a9cc2a105d7e64-fd45a0cf3a2d4657b02016bc60c5be34 received from {username='slave'} at 127.0.99.131:54275
I20240503 13:59:57.138664 8968 tablet_copy_service.cc:433] P 06ca8dbc004c42b99faa527a4e6f9545: ending tablet copy session 4a85e492c8154e9881a9cc2a105d7e64-fd45a0cf3a2d4657b02016bc60c5be34 on tablet fd45a0cf3a2d4657b02016bc60c5be34 with peer 4a85e492c8154e9881a9cc2a105d7e64
I20240503 13:59:57.170104 9081 consensus_queue.cc:237] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:57.171430 8946 raft_consensus.cc:1261] T ab6fe502a80c410a9b6ee6f5d3d3205e P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:57.171590 9343 raft_consensus.cc:1261] T ab6fe502a80c410a9b6ee6f5d3d3205e P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:57.171797 9015 consensus_peers.cc:485] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab6fe502a80c410a9b6ee6f5d3d3205e. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:57.171911 13280 consensus_queue.cc:1035] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:57.172446 13280 consensus_queue.cc:1035] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:57.173656 12864 raft_consensus.cc:2932] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:57.174021 8946 raft_consensus.cc:2932] T ab6fe502a80c410a9b6ee6f5d3d3205e P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:57.176468 9343 raft_consensus.cc:2932] T ab6fe502a80c410a9b6ee6f5d3d3205e P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:57.177224 6380 catalog_manager.cc:5484] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:57.262398 14603 ts_tablet_manager.cc:902] T af98fddcb623403c841a77ecc10a3e07 P 0d77752181e944698e809e325e5937e6: Initiating tablet copy from peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:57.262900 14603 tablet_copy_client.cc:308] T af98fddcb623403c841a77ecc10a3e07 P 0d77752181e944698e809e325e5937e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43531
I20240503 13:59:57.263411 6942 tablet_copy_service.cc:140] P f58065db70b64ef8b1dc499aa6ae4460: Received BeginTabletCopySession request for tablet af98fddcb623403c841a77ecc10a3e07 from peer 0d77752181e944698e809e325e5937e6 ({username='slave'} at 127.0.99.136:51497)
I20240503 13:59:57.263522 6942 tablet_copy_service.cc:161] P f58065db70b64ef8b1dc499aa6ae4460: Beginning new tablet copy session on tablet af98fddcb623403c841a77ecc10a3e07 from peer 0d77752181e944698e809e325e5937e6 at {username='slave'} at 127.0.99.136:51497: session id = 0d77752181e944698e809e325e5937e6-af98fddcb623403c841a77ecc10a3e07
I20240503 13:59:57.264441 6942 tablet_copy_source_session.cc:205] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:57.265286 14603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af98fddcb623403c841a77ecc10a3e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:57.267977 14603 tablet_copy_client.cc:783] T af98fddcb623403c841a77ecc10a3e07 P 0d77752181e944698e809e325e5937e6: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:57.268134 14603 tablet_copy_client.cc:647] T af98fddcb623403c841a77ecc10a3e07 P 0d77752181e944698e809e325e5937e6: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:57.269424 14603 tablet_copy_client.cc:523] T af98fddcb623403c841a77ecc10a3e07 P 0d77752181e944698e809e325e5937e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:57.270817 14603 tablet_bootstrap.cc:492] T af98fddcb623403c841a77ecc10a3e07 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:57.291584 14581 ts_tablet_manager.cc:902] T 038537d00e904bca89f0214d04fb4f13 P f58065db70b64ef8b1dc499aa6ae4460: Initiating tablet copy from peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:57.292145 14581 tablet_copy_client.cc:308] T 038537d00e904bca89f0214d04fb4f13 P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:34547
I20240503 13:59:57.292968 8968 tablet_copy_service.cc:140] P 06ca8dbc004c42b99faa527a4e6f9545: Received BeginTabletCopySession request for tablet 038537d00e904bca89f0214d04fb4f13 from peer f58065db70b64ef8b1dc499aa6ae4460 ({username='slave'} at 127.0.99.132:34657)
I20240503 13:59:57.293159 8968 tablet_copy_service.cc:161] P 06ca8dbc004c42b99faa527a4e6f9545: Beginning new tablet copy session on tablet 038537d00e904bca89f0214d04fb4f13 from peer f58065db70b64ef8b1dc499aa6ae4460 at {username='slave'} at 127.0.99.132:34657: session id = f58065db70b64ef8b1dc499aa6ae4460-038537d00e904bca89f0214d04fb4f13
I20240503 13:59:57.295043 8968 tablet_copy_source_session.cc:205] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:57.296489 14581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 038537d00e904bca89f0214d04fb4f13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:57.298982 14603 tablet_bootstrap.cc:492] T af98fddcb623403c841a77ecc10a3e07 P 0d77752181e944698e809e325e5937e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:57.299154 14581 tablet_copy_client.cc:783] T 038537d00e904bca89f0214d04fb4f13 P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:57.299371 14603 tablet_bootstrap.cc:492] T af98fddcb623403c841a77ecc10a3e07 P 0d77752181e944698e809e325e5937e6: Bootstrap complete.
I20240503 13:59:57.299432 14581 tablet_copy_client.cc:647] T 038537d00e904bca89f0214d04fb4f13 P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:57.299630 14867 ts_tablet_manager.cc:902] T b8c8bc1c29934e32a3f30ad20db05462 P 06ca8dbc004c42b99faa527a4e6f9545: Initiating tablet copy from peer 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:57.299623 14603 ts_tablet_manager.cc:1367] T af98fddcb623403c841a77ecc10a3e07 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.029s user 0.026s sys 0.002s
I20240503 13:59:57.300086 14603 raft_consensus.cc:348] T af98fddcb623403c841a77ecc10a3e07 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:57.300233 14603 raft_consensus.cc:729] T af98fddcb623403c841a77ecc10a3e07 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: LEARNER
I20240503 13:59:57.300357 14603 consensus_queue.cc:260] T af98fddcb623403c841a77ecc10a3e07 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:57.300541 14867 tablet_copy_client.cc:308] T b8c8bc1c29934e32a3f30ad20db05462 P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:39481
I20240503 13:59:57.300690 14603 ts_tablet_manager.cc:1398] T af98fddcb623403c841a77ecc10a3e07 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:57.301173 6942 tablet_copy_service.cc:342] P f58065db70b64ef8b1dc499aa6ae4460: Request end of tablet copy session 0d77752181e944698e809e325e5937e6-af98fddcb623403c841a77ecc10a3e07 received from {username='slave'} at 127.0.99.136:51497
I20240503 13:59:57.301252 6942 tablet_copy_service.cc:433] P f58065db70b64ef8b1dc499aa6ae4460: ending tablet copy session 0d77752181e944698e809e325e5937e6-af98fddcb623403c841a77ecc10a3e07 on tablet af98fddcb623403c841a77ecc10a3e07 with peer 0d77752181e944698e809e325e5937e6
I20240503 13:59:57.301545 14581 tablet_copy_client.cc:523] T 038537d00e904bca89f0214d04fb4f13 P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:57.301934 9101 tablet_copy_service.cc:140] P 8d574fb4a89d4eea947c859470d44dd4: Received BeginTabletCopySession request for tablet b8c8bc1c29934e32a3f30ad20db05462 from peer 06ca8dbc004c42b99faa527a4e6f9545 ({username='slave'} at 127.0.99.133:46917)
I20240503 13:59:57.302040 9101 tablet_copy_service.cc:161] P 8d574fb4a89d4eea947c859470d44dd4: Beginning new tablet copy session on tablet b8c8bc1c29934e32a3f30ad20db05462 from peer 06ca8dbc004c42b99faa527a4e6f9545 at {username='slave'} at 127.0.99.133:46917: session id = 06ca8dbc004c42b99faa527a4e6f9545-b8c8bc1c29934e32a3f30ad20db05462
I20240503 13:59:57.302829 14581 tablet_bootstrap.cc:492] T 038537d00e904bca89f0214d04fb4f13 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:57.302831 9101 tablet_copy_source_session.cc:205] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:57.307852 14867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8c8bc1c29934e32a3f30ad20db05462. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:57.309608 14867 tablet_copy_client.cc:783] T b8c8bc1c29934e32a3f30ad20db05462 P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:57.309767 14867 tablet_copy_client.cc:647] T b8c8bc1c29934e32a3f30ad20db05462 P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:57.310700 14867 tablet_copy_client.cc:523] T b8c8bc1c29934e32a3f30ad20db05462 P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:57.311679 14867 tablet_bootstrap.cc:492] T b8c8bc1c29934e32a3f30ad20db05462 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:57.313925 14581 tablet_bootstrap.cc:492] T 038537d00e904bca89f0214d04fb4f13 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:57.314189 14581 tablet_bootstrap.cc:492] T 038537d00e904bca89f0214d04fb4f13 P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap complete.
I20240503 13:59:57.314463 14581 ts_tablet_manager.cc:1367] T 038537d00e904bca89f0214d04fb4f13 P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240503 13:59:57.314867 14581 raft_consensus.cc:348] T 038537d00e904bca89f0214d04fb4f13 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:57.314996 14581 raft_consensus.cc:729] T 038537d00e904bca89f0214d04fb4f13 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: LEARNER
I20240503 13:59:57.315088 14581 consensus_queue.cc:260] T 038537d00e904bca89f0214d04fb4f13 P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:57.316458 14581 ts_tablet_manager.cc:1398] T 038537d00e904bca89f0214d04fb4f13 P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:57.318055 8968 tablet_copy_service.cc:342] P 06ca8dbc004c42b99faa527a4e6f9545: Request end of tablet copy session f58065db70b64ef8b1dc499aa6ae4460-038537d00e904bca89f0214d04fb4f13 received from {username='slave'} at 127.0.99.132:34657
I20240503 13:59:57.318125 8968 tablet_copy_service.cc:433] P 06ca8dbc004c42b99faa527a4e6f9545: ending tablet copy session f58065db70b64ef8b1dc499aa6ae4460-038537d00e904bca89f0214d04fb4f13 on tablet 038537d00e904bca89f0214d04fb4f13 with peer f58065db70b64ef8b1dc499aa6ae4460
I20240503 13:59:57.324713 14867 tablet_bootstrap.cc:492] T b8c8bc1c29934e32a3f30ad20db05462 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:57.324965 14867 tablet_bootstrap.cc:492] T b8c8bc1c29934e32a3f30ad20db05462 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap complete.
I20240503 13:59:57.325142 14867 ts_tablet_manager.cc:1367] T b8c8bc1c29934e32a3f30ad20db05462 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20240503 13:59:57.325567 14867 raft_consensus.cc:348] T b8c8bc1c29934e32a3f30ad20db05462 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 13:59:57.325696 14867 raft_consensus.cc:729] T b8c8bc1c29934e32a3f30ad20db05462 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: LEARNER
I20240503 13:59:57.325812 14867 consensus_queue.cc:260] T b8c8bc1c29934e32a3f30ad20db05462 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 13:59:57.326081 14867 ts_tablet_manager.cc:1398] T b8c8bc1c29934e32a3f30ad20db05462 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 13:59:57.328076 9101 tablet_copy_service.cc:342] P 8d574fb4a89d4eea947c859470d44dd4: Request end of tablet copy session 06ca8dbc004c42b99faa527a4e6f9545-b8c8bc1c29934e32a3f30ad20db05462 received from {username='slave'} at 127.0.99.133:46917
I20240503 13:59:57.328161 9101 tablet_copy_service.cc:433] P 8d574fb4a89d4eea947c859470d44dd4: ending tablet copy session 06ca8dbc004c42b99faa527a4e6f9545-b8c8bc1c29934e32a3f30ad20db05462 on tablet b8c8bc1c29934e32a3f30ad20db05462 with peer 06ca8dbc004c42b99faa527a4e6f9545
I20240503 13:59:57.345316 12367 raft_consensus.cc:1050] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545: attempting to promote NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 to VOTER
I20240503 13:59:57.345775 12367 consensus_queue.cc:237] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 13:59:57.347438 9210 raft_consensus.cc:1261] T 9f0daf41253f4678b158e3ccbb7b0c98 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:57.347867 14386 consensus_queue.cc:1035] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:57.348526 6788 raft_consensus.cc:1261] T 9f0daf41253f4678b158e3ccbb7b0c98 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:57.349426 14386 consensus_queue.cc:1035] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:57.350767 9343 raft_consensus.cc:1261] T 9f0daf41253f4678b158e3ccbb7b0c98 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:57.351229 14386 consensus_queue.cc:1035] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:57.351859 12032 raft_consensus.cc:1050] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64: attempting to promote NON_VOTER 2145d15ab2d74c0287839266f1411a43 to VOTER
I20240503 13:59:57.352211 12032 consensus_queue.cc:237] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 13:59:57.353296 14386 raft_consensus.cc:2932] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:57.353821 6526 raft_consensus.cc:1261] T 6ba45113776347c3adeae091c95427df P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:57.354403 6659 raft_consensus.cc:1261] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:57.354760 14880 consensus_queue.cc:1035] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:57.355463 9343 raft_consensus.cc:1261] T 6ba45113776347c3adeae091c95427df P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:57.355549 9342 raft_consensus.cc:2932] T 9f0daf41253f4678b158e3ccbb7b0c98 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:57.356698 14879 consensus_queue.cc:1035] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:57.356877 9210 raft_consensus.cc:2932] T 9f0daf41253f4678b158e3ccbb7b0c98 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:57.357046 14879 consensus_queue.cc:1035] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:57.357686 6379 catalog_manager.cc:5484] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:57.360491 6788 raft_consensus.cc:2932] T 9f0daf41253f4678b158e3ccbb7b0c98 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:57.364861 8946 consensus_queue.cc:237] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 13:59:57.366384 6788 raft_consensus.cc:1261] T 9f0daf41253f4678b158e3ccbb7b0c98 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:57.366467 9210 raft_consensus.cc:1261] T 9f0daf41253f4678b158e3ccbb7b0c98 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:57.366710 14384 consensus_queue.cc:1035] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:57.366866 14390 consensus_queue.cc:1035] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:57.367497 14879 raft_consensus.cc:2932] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:57.367980 9342 raft_consensus.cc:2932] T 6ba45113776347c3adeae091c95427df P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:57.369560 6659 raft_consensus.cc:2932] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:57.370113 14666 ts_tablet_manager.cc:902] T ce9410a4ed09491b8d37456ae44880a1 P ab1d21935d6049888eb0ceccc3c9d735: Initiating tablet copy from peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:57.370561 14666 tablet_copy_client.cc:308] T ce9410a4ed09491b8d37456ae44880a1 P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:46365
I20240503 13:59:57.371037 6811 tablet_copy_service.cc:140] P 4a85e492c8154e9881a9cc2a105d7e64: Received BeginTabletCopySession request for tablet ce9410a4ed09491b8d37456ae44880a1 from peer ab1d21935d6049888eb0ceccc3c9d735 ({username='slave'} at 127.0.99.135:47373)
I20240503 13:59:57.371129 6811 tablet_copy_service.cc:161] P 4a85e492c8154e9881a9cc2a105d7e64: Beginning new tablet copy session on tablet ce9410a4ed09491b8d37456ae44880a1 from peer ab1d21935d6049888eb0ceccc3c9d735 at {username='slave'} at 127.0.99.135:47373: session id = ab1d21935d6049888eb0ceccc3c9d735-ce9410a4ed09491b8d37456ae44880a1
I20240503 13:59:57.372009 6811 tablet_copy_source_session.cc:205] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:57.372052 6387 catalog_manager.cc:5484] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:57.372161 14390 raft_consensus.cc:2932] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:57.372965 6526 raft_consensus.cc:2932] T 6ba45113776347c3adeae091c95427df P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:57.374082 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9f0daf41253f4678b158e3ccbb7b0c98 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:57.374171 9210 raft_consensus.cc:2932] T 9f0daf41253f4678b158e3ccbb7b0c98 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:57.376030 6387 catalog_manager.cc:5484] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:57.376835 6790 consensus_queue.cc:237] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 13:59:57.377149 14666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce9410a4ed09491b8d37456ae44880a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:57.379057 9321 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f0daf41253f4678b158e3ccbb7b0c98 with delete_type TABLET_DATA_TOMBSTONED (TS 0d77752181e944698e809e325e5937e6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:57.380367 9343 raft_consensus.cc:1261] T 6ba45113776347c3adeae091c95427df P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:57.380563 6657 raft_consensus.cc:1261] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:57.380944 14879 consensus_queue.cc:1035] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:57.381667 14666 tablet_copy_client.cc:783] T ce9410a4ed09491b8d37456ae44880a1 P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:57.381819 14666 tablet_copy_client.cc:647] T ce9410a4ed09491b8d37456ae44880a1 P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:57.382877 14888 tablet_replica.cc:330] T 9f0daf41253f4678b158e3ccbb7b0c98 P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:57.383023 14888 raft_consensus.cc:2229] T 9f0daf41253f4678b158e3ccbb7b0c98 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:57.383184 14888 raft_consensus.cc:2258] T 9f0daf41253f4678b158e3ccbb7b0c98 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:57.383352 14879 consensus_queue.cc:1035] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:57.384696 14666 tablet_copy_client.cc:523] T ce9410a4ed09491b8d37456ae44880a1 P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:57.384881 14888 ts_tablet_manager.cc:1875] T 9f0daf41253f4678b158e3ccbb7b0c98 P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:57.386060 14666 tablet_bootstrap.cc:492] T ce9410a4ed09491b8d37456ae44880a1 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:57.388799 14880 raft_consensus.cc:2932] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:57.389043 6791 raft_consensus.cc:2932] T 9f0daf41253f4678b158e3ccbb7b0c98 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:57.389717 14888 ts_tablet_manager.cc:1888] T 9f0daf41253f4678b158e3ccbb7b0c98 P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:57.389828 14888 log.cc:1196] T 9f0daf41253f4678b158e3ccbb7b0c98 P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/9f0daf41253f4678b158e3ccbb7b0c98
I20240503 13:59:57.391049 6372 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6ba45113776347c3adeae091c95427df with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:57.391155 9343 raft_consensus.cc:2932] T 6ba45113776347c3adeae091c95427df P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:57.391156 6658 raft_consensus.cc:2932] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:57.391341 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet 9f0daf41253f4678b158e3ccbb7b0c98 (table rebalancer_extra_table_36 [id=25d54e91bed8402eb84d2c36a0601777]) successfully deleted
I20240503 13:59:57.394109 14603 ts_tablet_manager.cc:902] T f403dc06195d449e8f454a9f4e795925 P 0d77752181e944698e809e325e5937e6: Initiating tablet copy from peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:57.394385 6384 catalog_manager.cc:5484] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index 3 to 4, VOTER 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:57.394568 14603 tablet_copy_client.cc:308] T f403dc06195d449e8f454a9f4e795925 P 0d77752181e944698e809e325e5937e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43531
I20240503 13:59:57.394992 6942 tablet_copy_service.cc:140] P f58065db70b64ef8b1dc499aa6ae4460: Received BeginTabletCopySession request for tablet f403dc06195d449e8f454a9f4e795925 from peer 0d77752181e944698e809e325e5937e6 ({username='slave'} at 127.0.99.136:51497)
I20240503 13:59:57.395097 6942 tablet_copy_service.cc:161] P f58065db70b64ef8b1dc499aa6ae4460: Beginning new tablet copy session on tablet f403dc06195d449e8f454a9f4e795925 from peer 0d77752181e944698e809e325e5937e6 at {username='slave'} at 127.0.99.136:51497: session id = 0d77752181e944698e809e325e5937e6-f403dc06195d449e8f454a9f4e795925
I20240503 13:59:57.396121 6942 tablet_copy_source_session.cc:205] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:57.397092 6507 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ba45113776347c3adeae091c95427df with delete_type TABLET_DATA_TOMBSTONED (TS 5f46e19644fd4f4b825c86ba6af768da not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40142
I20240503 13:59:57.397632 14603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f403dc06195d449e8f454a9f4e795925. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:57.399585 14603 tablet_copy_client.cc:783] T f403dc06195d449e8f454a9f4e795925 P 0d77752181e944698e809e325e5937e6: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:57.399722 14603 tablet_copy_client.cc:647] T f403dc06195d449e8f454a9f4e795925 P 0d77752181e944698e809e325e5937e6: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:57.401556 14603 tablet_copy_client.cc:523] T f403dc06195d449e8f454a9f4e795925 P 0d77752181e944698e809e325e5937e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:57.402324 14893 tablet_replica.cc:330] T 6ba45113776347c3adeae091c95427df P 5f46e19644fd4f4b825c86ba6af768da: stopping tablet replica
I20240503 13:59:57.402487 14893 raft_consensus.cc:2229] T 6ba45113776347c3adeae091c95427df P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:57.402525 14603 tablet_bootstrap.cc:492] T f403dc06195d449e8f454a9f4e795925 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:57.402642 14893 raft_consensus.cc:2258] T 6ba45113776347c3adeae091c95427df P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:57.403945 14893 ts_tablet_manager.cc:1875] T 6ba45113776347c3adeae091c95427df P 5f46e19644fd4f4b825c86ba6af768da: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:57.406113 14603 tablet_bootstrap.cc:492] T f403dc06195d449e8f454a9f4e795925 P 0d77752181e944698e809e325e5937e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:57.406356 14603 tablet_bootstrap.cc:492] T f403dc06195d449e8f454a9f4e795925 P 0d77752181e944698e809e325e5937e6: Bootstrap complete.
I20240503 13:59:57.406508 14603 ts_tablet_manager.cc:1367] T f403dc06195d449e8f454a9f4e795925 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 13:59:57.406917 14603 raft_consensus.cc:348] T f403dc06195d449e8f454a9f4e795925 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:57.407053 14603 raft_consensus.cc:729] T f403dc06195d449e8f454a9f4e795925 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: LEARNER
I20240503 13:59:57.407159 14603 consensus_queue.cc:260] T f403dc06195d449e8f454a9f4e795925 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:57.407521 14603 ts_tablet_manager.cc:1398] T f403dc06195d449e8f454a9f4e795925 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:57.407640 14893 ts_tablet_manager.cc:1888] T 6ba45113776347c3adeae091c95427df P 5f46e19644fd4f4b825c86ba6af768da: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:57.407747 14893 log.cc:1196] T 6ba45113776347c3adeae091c95427df P 5f46e19644fd4f4b825c86ba6af768da: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/6ba45113776347c3adeae091c95427df
I20240503 13:59:57.408140 6942 tablet_copy_service.cc:342] P f58065db70b64ef8b1dc499aa6ae4460: Request end of tablet copy session 0d77752181e944698e809e325e5937e6-f403dc06195d449e8f454a9f4e795925 received from {username='slave'} at 127.0.99.136:51497
I20240503 13:59:57.408233 6942 tablet_copy_service.cc:433] P f58065db70b64ef8b1dc499aa6ae4460: ending tablet copy session 0d77752181e944698e809e325e5937e6-f403dc06195d449e8f454a9f4e795925 on tablet f403dc06195d449e8f454a9f4e795925 with peer 0d77752181e944698e809e325e5937e6
I20240503 13:59:57.408902 6373 catalog_manager.cc:4830] TS 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021): tablet 6ba45113776347c3adeae091c95427df (table rebalancer_extra_table_72 [id=fb05f2e903bb419db0bfe7d54f35b083]) successfully deleted
I20240503 13:59:57.409293 6922 raft_consensus.cc:1203] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:57.416117 14867 ts_tablet_manager.cc:902] T efe57c7f306b4b0ab934388f5a372a74 P 06ca8dbc004c42b99faa527a4e6f9545: Initiating tablet copy from peer 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:57.416224 14896 ts_tablet_manager.cc:902] T 174e5b7577d94d87b729d1735987a97a P 8d574fb4a89d4eea947c859470d44dd4: Initiating tablet copy from peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:57.416574 14867 tablet_copy_client.cc:308] T efe57c7f306b4b0ab934388f5a372a74 P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46119
I20240503 13:59:57.416747 14896 tablet_copy_client.cc:308] T 174e5b7577d94d87b729d1735987a97a P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:34547
I20240503 13:59:57.416966 6680 tablet_copy_service.cc:140] P 2145d15ab2d74c0287839266f1411a43: Received BeginTabletCopySession request for tablet efe57c7f306b4b0ab934388f5a372a74 from peer 06ca8dbc004c42b99faa527a4e6f9545 ({username='slave'} at 127.0.99.133:36361)
I20240503 13:59:57.417069 6680 tablet_copy_service.cc:161] P 2145d15ab2d74c0287839266f1411a43: Beginning new tablet copy session on tablet efe57c7f306b4b0ab934388f5a372a74 from peer 06ca8dbc004c42b99faa527a4e6f9545 at {username='slave'} at 127.0.99.133:36361: session id = 06ca8dbc004c42b99faa527a4e6f9545-efe57c7f306b4b0ab934388f5a372a74
I20240503 13:59:57.417294 8968 tablet_copy_service.cc:140] P 06ca8dbc004c42b99faa527a4e6f9545: Received BeginTabletCopySession request for tablet 174e5b7577d94d87b729d1735987a97a from peer 8d574fb4a89d4eea947c859470d44dd4 ({username='slave'} at 127.0.99.134:48775)
I20240503 13:59:57.417380 8968 tablet_copy_service.cc:161] P 06ca8dbc004c42b99faa527a4e6f9545: Beginning new tablet copy session on tablet 174e5b7577d94d87b729d1735987a97a from peer 8d574fb4a89d4eea947c859470d44dd4 at {username='slave'} at 127.0.99.134:48775: session id = 8d574fb4a89d4eea947c859470d44dd4-174e5b7577d94d87b729d1735987a97a
I20240503 13:59:57.417986 6680 tablet_copy_source_session.cc:205] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:57.418229 8968 tablet_copy_source_session.cc:205] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:57.418802 14867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efe57c7f306b4b0ab934388f5a372a74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:57.419916 14896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174e5b7577d94d87b729d1735987a97a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:57.420964 14867 tablet_copy_client.cc:783] T efe57c7f306b4b0ab934388f5a372a74 P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:57.421104 14867 tablet_copy_client.cc:647] T efe57c7f306b4b0ab934388f5a372a74 P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:57.422627 14867 tablet_copy_client.cc:523] T efe57c7f306b4b0ab934388f5a372a74 P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:57.423952 14867 tablet_bootstrap.cc:492] T efe57c7f306b4b0ab934388f5a372a74 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:57.427248 14896 tablet_copy_client.cc:783] T 174e5b7577d94d87b729d1735987a97a P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:57.427428 14896 tablet_copy_client.cc:647] T 174e5b7577d94d87b729d1735987a97a P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:57.428320 14896 tablet_copy_client.cc:523] T 174e5b7577d94d87b729d1735987a97a P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:57.429332 14666 tablet_bootstrap.cc:492] T ce9410a4ed09491b8d37456ae44880a1 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:57.429371 14896 tablet_bootstrap.cc:492] T 174e5b7577d94d87b729d1735987a97a P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:57.429594 14666 tablet_bootstrap.cc:492] T ce9410a4ed09491b8d37456ae44880a1 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap complete.
I20240503 13:59:57.429750 14666 ts_tablet_manager.cc:1367] T ce9410a4ed09491b8d37456ae44880a1 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.044s user 0.024s sys 0.000s
I20240503 13:59:57.430146 14666 raft_consensus.cc:348] T ce9410a4ed09491b8d37456ae44880a1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:57.430280 14666 raft_consensus.cc:729] T ce9410a4ed09491b8d37456ae44880a1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: LEARNER
I20240503 13:59:57.430383 14666 consensus_queue.cc:260] T ce9410a4ed09491b8d37456ae44880a1 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:57.430647 14666 ts_tablet_manager.cc:1398] T ce9410a4ed09491b8d37456ae44880a1 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:57.433002 14896 tablet_bootstrap.cc:492] T 174e5b7577d94d87b729d1735987a97a P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:57.433274 14896 tablet_bootstrap.cc:492] T 174e5b7577d94d87b729d1735987a97a P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap complete.
I20240503 13:59:57.433425 14896 ts_tablet_manager.cc:1367] T 174e5b7577d94d87b729d1735987a97a P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20240503 13:59:57.433967 14896 raft_consensus.cc:348] T 174e5b7577d94d87b729d1735987a97a P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:57.434095 14896 raft_consensus.cc:729] T 174e5b7577d94d87b729d1735987a97a P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: LEARNER
I20240503 13:59:57.434214 14896 consensus_queue.cc:260] T 174e5b7577d94d87b729d1735987a97a P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:57.434312 6811 tablet_copy_service.cc:342] P 4a85e492c8154e9881a9cc2a105d7e64: Request end of tablet copy session ab1d21935d6049888eb0ceccc3c9d735-ce9410a4ed09491b8d37456ae44880a1 received from {username='slave'} at 127.0.99.135:47373
I20240503 13:59:57.434381 6811 tablet_copy_service.cc:433] P 4a85e492c8154e9881a9cc2a105d7e64: ending tablet copy session ab1d21935d6049888eb0ceccc3c9d735-ce9410a4ed09491b8d37456ae44880a1 on tablet ce9410a4ed09491b8d37456ae44880a1 with peer ab1d21935d6049888eb0ceccc3c9d735
I20240503 13:59:57.434525 14896 ts_tablet_manager.cc:1398] T 174e5b7577d94d87b729d1735987a97a P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 13:59:57.435200 8968 tablet_copy_service.cc:342] P 06ca8dbc004c42b99faa527a4e6f9545: Request end of tablet copy session 8d574fb4a89d4eea947c859470d44dd4-174e5b7577d94d87b729d1735987a97a received from {username='slave'} at 127.0.99.134:48775
I20240503 13:59:57.435272 8968 tablet_copy_service.cc:433] P 06ca8dbc004c42b99faa527a4e6f9545: ending tablet copy session 8d574fb4a89d4eea947c859470d44dd4-174e5b7577d94d87b729d1735987a97a on tablet 174e5b7577d94d87b729d1735987a97a with peer 8d574fb4a89d4eea947c859470d44dd4
I20240503 13:59:57.440970 9343 raft_consensus.cc:1203] T 3612cd35d8fa4300a1132fe43a63e6c1 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:57.451699 14867 tablet_bootstrap.cc:492] T efe57c7f306b4b0ab934388f5a372a74 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:57.452044 14867 tablet_bootstrap.cc:492] T efe57c7f306b4b0ab934388f5a372a74 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap complete.
I20240503 13:59:57.452253 14867 ts_tablet_manager.cc:1367] T efe57c7f306b4b0ab934388f5a372a74 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.028s user 0.023s sys 0.002s
I20240503 13:59:57.452733 14867 raft_consensus.cc:348] T efe57c7f306b4b0ab934388f5a372a74 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 13:59:57.452872 14867 raft_consensus.cc:729] T efe57c7f306b4b0ab934388f5a372a74 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: LEARNER
I20240503 13:59:57.452981 14867 consensus_queue.cc:260] T efe57c7f306b4b0ab934388f5a372a74 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 13:59:57.453291 14867 ts_tablet_manager.cc:1398] T efe57c7f306b4b0ab934388f5a372a74 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:57.453847 6680 tablet_copy_service.cc:342] P 2145d15ab2d74c0287839266f1411a43: Request end of tablet copy session 06ca8dbc004c42b99faa527a4e6f9545-efe57c7f306b4b0ab934388f5a372a74 received from {username='slave'} at 127.0.99.133:36361
I20240503 13:59:57.453944 6680 tablet_copy_service.cc:433] P 2145d15ab2d74c0287839266f1411a43: ending tablet copy session 06ca8dbc004c42b99faa527a4e6f9545-efe57c7f306b4b0ab934388f5a372a74 on tablet efe57c7f306b4b0ab934388f5a372a74 with peer 06ca8dbc004c42b99faa527a4e6f9545
I20240503 13:59:57.487236 6526 raft_consensus.cc:1203] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:57.488173 9210 raft_consensus.cc:1203] T 7c46822ccc1e48469c8d7f31b0bb0b8a P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:57.500644 6922 raft_consensus.cc:1203] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:57.510567 14666 ts_tablet_manager.cc:902] T 3331108688a3485a9c44c65600f0f3e5 P ab1d21935d6049888eb0ceccc3c9d735: Initiating tablet copy from peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:57.511006 14666 tablet_copy_client.cc:308] T 3331108688a3485a9c44c65600f0f3e5 P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43531
I20240503 13:59:57.511909 6942 tablet_copy_service.cc:140] P f58065db70b64ef8b1dc499aa6ae4460: Received BeginTabletCopySession request for tablet 3331108688a3485a9c44c65600f0f3e5 from peer ab1d21935d6049888eb0ceccc3c9d735 ({username='slave'} at 127.0.99.135:34893)
I20240503 13:59:57.512001 6942 tablet_copy_service.cc:161] P f58065db70b64ef8b1dc499aa6ae4460: Beginning new tablet copy session on tablet 3331108688a3485a9c44c65600f0f3e5 from peer ab1d21935d6049888eb0ceccc3c9d735 at {username='slave'} at 127.0.99.135:34893: session id = ab1d21935d6049888eb0ceccc3c9d735-3331108688a3485a9c44c65600f0f3e5
I20240503 13:59:57.512899 6942 tablet_copy_source_session.cc:205] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:57.514673 14666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3331108688a3485a9c44c65600f0f3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:57.516971 14666 tablet_copy_client.cc:783] T 3331108688a3485a9c44c65600f0f3e5 P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:57.517134 14666 tablet_copy_client.cc:647] T 3331108688a3485a9c44c65600f0f3e5 P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:57.519188 14666 tablet_copy_client.cc:523] T 3331108688a3485a9c44c65600f0f3e5 P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:57.520241 14666 tablet_bootstrap.cc:492] T 3331108688a3485a9c44c65600f0f3e5 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:57.541877 14666 tablet_bootstrap.cc:492] T 3331108688a3485a9c44c65600f0f3e5 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:57.542160 14666 tablet_bootstrap.cc:492] T 3331108688a3485a9c44c65600f0f3e5 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap complete.
I20240503 13:59:57.542351 14666 ts_tablet_manager.cc:1367] T 3331108688a3485a9c44c65600f0f3e5 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.005s
I20240503 13:59:57.542759 14666 raft_consensus.cc:348] T 3331108688a3485a9c44c65600f0f3e5 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:57.542869 14666 raft_consensus.cc:729] T 3331108688a3485a9c44c65600f0f3e5 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: LEARNER
I20240503 13:59:57.543116 14666 consensus_queue.cc:260] T 3331108688a3485a9c44c65600f0f3e5 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:57.543437 14666 ts_tablet_manager.cc:1398] T 3331108688a3485a9c44c65600f0f3e5 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.000s sys 0.003s
I20240503 13:59:57.544040 6942 tablet_copy_service.cc:342] P f58065db70b64ef8b1dc499aa6ae4460: Request end of tablet copy session ab1d21935d6049888eb0ceccc3c9d735-3331108688a3485a9c44c65600f0f3e5 received from {username='slave'} at 127.0.99.135:34893
I20240503 13:59:57.544121 6942 tablet_copy_service.cc:433] P f58065db70b64ef8b1dc499aa6ae4460: ending tablet copy session ab1d21935d6049888eb0ceccc3c9d735-3331108688a3485a9c44c65600f0f3e5 on tablet 3331108688a3485a9c44c65600f0f3e5 with peer ab1d21935d6049888eb0ceccc3c9d735
I20240503 13:59:57.560351 6920 raft_consensus.cc:1203] T f16b677f47f54a13a7f374cce34713bd P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:57.609527 6788 raft_consensus.cc:1203] T fd45a0cf3a2d4657b02016bc60c5be34 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:57.687228 6527 raft_consensus.cc:1203] T 63d2b0a8c3734720ba7d72a36d344cba P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:57.705673 15053 ts_tablet_manager.cc:902] T ab6fe502a80c410a9b6ee6f5d3d3205e P 4a85e492c8154e9881a9cc2a105d7e64: Initiating tablet copy from peer 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:57.706171 15053 tablet_copy_client.cc:308] T ab6fe502a80c410a9b6ee6f5d3d3205e P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:39481
I20240503 13:59:57.706665 9101 tablet_copy_service.cc:140] P 8d574fb4a89d4eea947c859470d44dd4: Received BeginTabletCopySession request for tablet ab6fe502a80c410a9b6ee6f5d3d3205e from peer 4a85e492c8154e9881a9cc2a105d7e64 ({username='slave'} at 127.0.99.131:44581)
I20240503 13:59:57.706779 9101 tablet_copy_service.cc:161] P 8d574fb4a89d4eea947c859470d44dd4: Beginning new tablet copy session on tablet ab6fe502a80c410a9b6ee6f5d3d3205e from peer 4a85e492c8154e9881a9cc2a105d7e64 at {username='slave'} at 127.0.99.131:44581: session id = 4a85e492c8154e9881a9cc2a105d7e64-ab6fe502a80c410a9b6ee6f5d3d3205e
I20240503 13:59:57.707687 9101 tablet_copy_source_session.cc:205] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:57.709077 15053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab6fe502a80c410a9b6ee6f5d3d3205e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:57.711393 15053 tablet_copy_client.cc:783] T ab6fe502a80c410a9b6ee6f5d3d3205e P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:57.711560 15053 tablet_copy_client.cc:647] T ab6fe502a80c410a9b6ee6f5d3d3205e P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:57.712584 15053 tablet_copy_client.cc:523] T ab6fe502a80c410a9b6ee6f5d3d3205e P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:57.713769 15053 tablet_bootstrap.cc:492] T ab6fe502a80c410a9b6ee6f5d3d3205e P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:57.717483 15053 tablet_bootstrap.cc:492] T ab6fe502a80c410a9b6ee6f5d3d3205e P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:57.717695 15053 tablet_bootstrap.cc:492] T ab6fe502a80c410a9b6ee6f5d3d3205e P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap complete.
I20240503 13:59:57.717839 15053 ts_tablet_manager.cc:1367] T ab6fe502a80c410a9b6ee6f5d3d3205e P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 13:59:57.718294 15053 raft_consensus.cc:348] T ab6fe502a80c410a9b6ee6f5d3d3205e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:57.718539 15053 raft_consensus.cc:729] T ab6fe502a80c410a9b6ee6f5d3d3205e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: LEARNER
I20240503 13:59:57.718652 15053 consensus_queue.cc:260] T ab6fe502a80c410a9b6ee6f5d3d3205e P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:57.718925 15053 ts_tablet_manager.cc:1398] T ab6fe502a80c410a9b6ee6f5d3d3205e P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:57.719687 9101 tablet_copy_service.cc:342] P 8d574fb4a89d4eea947c859470d44dd4: Request end of tablet copy session 4a85e492c8154e9881a9cc2a105d7e64-ab6fe502a80c410a9b6ee6f5d3d3205e received from {username='slave'} at 127.0.99.131:44581
I20240503 13:59:57.719774 9101 tablet_copy_service.cc:433] P 8d574fb4a89d4eea947c859470d44dd4: ending tablet copy session 4a85e492c8154e9881a9cc2a105d7e64-ab6fe502a80c410a9b6ee6f5d3d3205e on tablet ab6fe502a80c410a9b6ee6f5d3d3205e with peer 4a85e492c8154e9881a9cc2a105d7e64
I20240503 13:59:57.749236 8946 raft_consensus.cc:1203] T b8c8bc1c29934e32a3f30ad20db05462 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:57.855846 6922 raft_consensus.cc:1203] T 038537d00e904bca89f0214d04fb4f13 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:57.866006 9343 raft_consensus.cc:1203] T af98fddcb623403c841a77ecc10a3e07 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:57.869614 14177 raft_consensus.cc:1050] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43: attempting to promote NON_VOTER 0d77752181e944698e809e325e5937e6 to VOTER
I20240503 13:59:57.870061 14177 consensus_queue.cc:237] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 13:59:57.871230 9343 raft_consensus.cc:1261] T 3612cd35d8fa4300a1132fe43a63e6c1 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:57.871307 6922 raft_consensus.cc:1261] T 3612cd35d8fa4300a1132fe43a63e6c1 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:57.871629 6791 raft_consensus.cc:1261] T 3612cd35d8fa4300a1132fe43a63e6c1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:57.871625 12915 consensus_queue.cc:1035] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:57.872082 14177 consensus_queue.cc:1035] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:57.872246 12915 consensus_queue.cc:1035] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:57.874279 14177 raft_consensus.cc:2932] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:57.874668 6791 raft_consensus.cc:2932] T 3612cd35d8fa4300a1132fe43a63e6c1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:57.877796 6922 raft_consensus.cc:2932] T 3612cd35d8fa4300a1132fe43a63e6c1 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:57.878074 6380 catalog_manager.cc:5484] T 3612cd35d8fa4300a1132fe43a63e6c1 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:57.879832 9342 raft_consensus.cc:2932] T 3612cd35d8fa4300a1132fe43a63e6c1 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:57.883980 9210 raft_consensus.cc:1203] T ce9410a4ed09491b8d37456ae44880a1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:57.885644 9342 raft_consensus.cc:1203] T f403dc06195d449e8f454a9f4e795925 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:57.886540 6658 consensus_queue.cc:237] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 13:59:57.888109 6922 raft_consensus.cc:1261] T 3612cd35d8fa4300a1132fe43a63e6c1 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:57.888556 9343 raft_consensus.cc:1261] T 3612cd35d8fa4300a1132fe43a63e6c1 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:57.888541 14907 consensus_queue.cc:1035] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:57.888927 14907 consensus_queue.cc:1035] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:57.890885 12915 raft_consensus.cc:2932] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:57.891544 6922 raft_consensus.cc:2932] T 3612cd35d8fa4300a1132fe43a63e6c1 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:57.892789 9343 raft_consensus.cc:2932] T 3612cd35d8fa4300a1132fe43a63e6c1 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:57.893035 13159 raft_consensus.cc:1050] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460: attempting to promote NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 to VOTER
I20240503 13:59:57.893520 13159 consensus_queue.cc:237] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 13:59:57.894835 6374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3612cd35d8fa4300a1132fe43a63e6c1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:57.895124 6527 raft_consensus.cc:1261] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:57.895217 9210 raft_consensus.cc:1261] T 7c46822ccc1e48469c8d7f31b0bb0b8a P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:57.895254 6658 raft_consensus.cc:1261] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:57.895538 13370 consensus_queue.cc:1035] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:57.895874 13370 consensus_queue.cc:1035] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:57.896165 13370 consensus_queue.cc:1035] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:57.896198 6379 catalog_manager.cc:5484] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index 5 to 6, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:57.899847 13159 raft_consensus.cc:2932] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:57.900756 6769 tablet_service.cc:1514] Processing DeleteTablet for tablet 3612cd35d8fa4300a1132fe43a63e6c1 with delete_type TABLET_DATA_TOMBSTONED (TS 4a85e492c8154e9881a9cc2a105d7e64 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:57.901172 15109 tablet_replica.cc:330] T 3612cd35d8fa4300a1132fe43a63e6c1 P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:57.901355 15109 raft_consensus.cc:2229] T 3612cd35d8fa4300a1132fe43a63e6c1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:57.901546 15109 raft_consensus.cc:2258] T 3612cd35d8fa4300a1132fe43a63e6c1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:57.902408 6527 raft_consensus.cc:2932] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:57.902451 6658 raft_consensus.cc:2932] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:57.902806 15109 ts_tablet_manager.cc:1875] T 3612cd35d8fa4300a1132fe43a63e6c1 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:57.905026 9210 raft_consensus.cc:2932] T 7c46822ccc1e48469c8d7f31b0bb0b8a P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:57.905149 6384 catalog_manager.cc:5484] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:57.905825 15109 ts_tablet_manager.cc:1888] T 3612cd35d8fa4300a1132fe43a63e6c1 P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:57.905939 15109 log.cc:1196] T 3612cd35d8fa4300a1132fe43a63e6c1 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/3612cd35d8fa4300a1132fe43a63e6c1
I20240503 13:59:57.906569 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet 3612cd35d8fa4300a1132fe43a63e6c1 (table rebalance_test_table_19 [id=aaf095359a4a40ecbae916f4245a8f5f]) successfully deleted
I20240503 13:59:57.911123 6922 consensus_queue.cc:237] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 13:59:57.912354 9210 raft_consensus.cc:1261] T 7c46822ccc1e48469c8d7f31b0bb0b8a P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:57.912775 15110 consensus_queue.cc:1035] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:57.913480 6527 raft_consensus.cc:1261] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:57.914076 15110 consensus_queue.cc:1035] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:57.914547 15110 raft_consensus.cc:2932] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:57.914930 9210 raft_consensus.cc:2932] T 7c46822ccc1e48469c8d7f31b0bb0b8a P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:57.916226 6527 raft_consensus.cc:2932] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:57.917039 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7c46822ccc1e48469c8d7f31b0bb0b8a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:57.917963 6384 catalog_manager.cc:5484] T 7c46822ccc1e48469c8d7f31b0bb0b8a P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:57.922133 6639 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c46822ccc1e48469c8d7f31b0bb0b8a with delete_type TABLET_DATA_TOMBSTONED (TS 2145d15ab2d74c0287839266f1411a43 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:57.923193 15112 tablet_replica.cc:330] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:57.923786 15112 raft_consensus.cc:2229] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:57.924651 15112 raft_consensus.cc:2258] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:57.925488 15112 ts_tablet_manager.cc:1875] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:57.927615 15112 ts_tablet_manager.cc:1888] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:57.927727 15112 log.cc:1196] T 7c46822ccc1e48469c8d7f31b0bb0b8a P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/7c46822ccc1e48469c8d7f31b0bb0b8a
I20240503 13:59:57.928284 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet 7c46822ccc1e48469c8d7f31b0bb0b8a (table rebalance_test_table_9 [id=010fdec94fd24d33b4513ebe788694ac]) successfully deleted
I20240503 13:59:57.982829 12372 raft_consensus.cc:1050] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4: attempting to promote NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 to VOTER
I20240503 13:59:57.983309 12372 consensus_queue.cc:237] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 13:59:57.984855 8946 raft_consensus.cc:1261] T f16b677f47f54a13a7f374cce34713bd P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:57.984963 6922 raft_consensus.cc:1261] T f16b677f47f54a13a7f374cce34713bd P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:57.984987 6658 raft_consensus.cc:1261] T f16b677f47f54a13a7f374cce34713bd P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:57.985428 15124 consensus_queue.cc:1035] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:57.986037 13280 consensus_queue.cc:1035] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:57.986162 15124 consensus_queue.cc:1035] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:57.989295 12372 raft_consensus.cc:2932] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:57.990058 8946 raft_consensus.cc:2932] T f16b677f47f54a13a7f374cce34713bd P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:57.991147 6658 raft_consensus.cc:2932] T f16b677f47f54a13a7f374cce34713bd P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:57.994163 6922 raft_consensus.cc:2932] T f16b677f47f54a13a7f374cce34713bd P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:57.994315 6379 catalog_manager.cc:5484] T f16b677f47f54a13a7f374cce34713bd P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:57.998538 9081 consensus_queue.cc:237] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 13:59:58.001166 6658 raft_consensus.cc:1261] T f16b677f47f54a13a7f374cce34713bd P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.001510 13280 consensus_queue.cc:1035] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.001552 6922 raft_consensus.cc:1261] T f16b677f47f54a13a7f374cce34713bd P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.002063 13280 consensus_queue.cc:1035] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.003999 14146 raft_consensus.cc:2932] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:58.005662 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f16b677f47f54a13a7f374cce34713bd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:58.006060 6922 raft_consensus.cc:2932] T f16b677f47f54a13a7f374cce34713bd P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:58.007616 6387 catalog_manager.cc:5484] T f16b677f47f54a13a7f374cce34713bd P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:58.010773 12372 raft_consensus.cc:1050] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4: attempting to promote NON_VOTER 5f46e19644fd4f4b825c86ba6af768da to VOTER
I20240503 13:59:58.011207 12372 consensus_queue.cc:237] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 13:59:58.011837 6658 raft_consensus.cc:2932] T f16b677f47f54a13a7f374cce34713bd P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:58.012671 8926 tablet_service.cc:1514] Processing DeleteTablet for tablet f16b677f47f54a13a7f374cce34713bd with delete_type TABLET_DATA_TOMBSTONED (TS 06ca8dbc004c42b99faa527a4e6f9545 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:58.013295 6791 raft_consensus.cc:1261] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.013285 8946 raft_consensus.cc:1203] T efe57c7f306b4b0ab934388f5a372a74 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:58.013716 15124 consensus_queue.cc:1035] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.014763 6527 raft_consensus.cc:1261] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.014904 9210 raft_consensus.cc:1261] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.016739 15124 consensus_queue.cc:1035] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.017072 12372 consensus_queue.cc:1035] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.018370 15143 tablet_replica.cc:330] T f16b677f47f54a13a7f374cce34713bd P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:58.018532 15143 raft_consensus.cc:2229] T f16b677f47f54a13a7f374cce34713bd P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:58.018764 12372 raft_consensus.cc:2932] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:58.019685 6527 raft_consensus.cc:2932] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:58.019690 9210 raft_consensus.cc:2932] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:58.020843 6788 raft_consensus.cc:2932] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:58.023802 15143 raft_consensus.cc:2258] T f16b677f47f54a13a7f374cce34713bd P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:58.024096 6380 catalog_manager.cc:5484] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:58.024463 15143 ts_tablet_manager.cc:1875] T f16b677f47f54a13a7f374cce34713bd P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:58.027734 15143 ts_tablet_manager.cc:1888] T f16b677f47f54a13a7f374cce34713bd P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:58.028164 15143 log.cc:1196] T f16b677f47f54a13a7f374cce34713bd P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/f16b677f47f54a13a7f374cce34713bd
I20240503 13:59:58.028828 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet f16b677f47f54a13a7f374cce34713bd (table rebalancer_extra_table_62 [id=550b2587d1ad46df98279b7c7b5fa594]) successfully deleted
I20240503 13:59:58.030097 9081 raft_consensus.cc:1203] T 174e5b7577d94d87b729d1735987a97a P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:58.030511 14880 raft_consensus.cc:1050] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64: attempting to promote NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 to VOTER
I20240503 13:59:58.030882 14880 consensus_queue.cc:237] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 13:59:58.032354 6922 raft_consensus.cc:1261] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.033217 9081 raft_consensus.cc:1261] T b5780f4a85b146e2bfbbf7d2feaca316 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.034128 9210 raft_consensus.cc:1261] T b5780f4a85b146e2bfbbf7d2feaca316 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.034417 14879 consensus_queue.cc:1035] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.034713 12032 consensus_queue.cc:1035] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.035531 15149 consensus_queue.cc:1035] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.041440 12032 raft_consensus.cc:2932] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:58.042227 9210 raft_consensus.cc:1203] T 3331108688a3485a9c44c65600f0f3e5 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:58.044788 9081 raft_consensus.cc:2932] T b5780f4a85b146e2bfbbf7d2feaca316 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:58.044834 6922 raft_consensus.cc:2932] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:58.045111 6379 catalog_manager.cc:5484] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:58.055176 14384 raft_consensus.cc:1050] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545: attempting to promote NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 to VOTER
I20240503 13:59:58.055577 14384 consensus_queue.cc:237] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 13:59:58.056937 9343 raft_consensus.cc:1261] T fd45a0cf3a2d4657b02016bc60c5be34 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.057178 6788 raft_consensus.cc:1261] T fd45a0cf3a2d4657b02016bc60c5be34 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.057653 12367 consensus_queue.cc:1035] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.057807 14390 consensus_queue.cc:1035] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.059517 6788 consensus_queue.cc:237] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 13:59:58.059854 9081 raft_consensus.cc:1261] T fd45a0cf3a2d4657b02016bc60c5be34 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.060359 9210 raft_consensus.cc:2932] T b5780f4a85b146e2bfbbf7d2feaca316 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:58.062440 14390 consensus_queue.cc:1035] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.062631 6922 raft_consensus.cc:1261] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.063162 15149 consensus_queue.cc:1035] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.067576 9205 raft_consensus.cc:1261] T b5780f4a85b146e2bfbbf7d2feaca316 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.068918 12032 consensus_queue.cc:1035] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.069454 12032 raft_consensus.cc:2932] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:58.071728 6920 raft_consensus.cc:2932] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:58.072522 6387 catalog_manager.cc:5484] T b5780f4a85b146e2bfbbf7d2feaca316 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:58.073329 6372 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b5780f4a85b146e2bfbbf7d2feaca316 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:58.074213 11378 raft_consensus.cc:1050] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6: attempting to promote NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 to VOTER
I20240503 13:59:58.074613 11378 consensus_queue.cc:237] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 13:59:58.076457 9210 raft_consensus.cc:1261] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.076524 8946 raft_consensus.cc:1261] T 1e5875904e6047eb82f28c3cbc9318fc P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.076742 9060 tablet_service.cc:1514] Processing DeleteTablet for tablet b5780f4a85b146e2bfbbf7d2feaca316 with delete_type TABLET_DATA_TOMBSTONED (TS 8d574fb4a89d4eea947c859470d44dd4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:58.076886 14432 consensus_queue.cc:1035] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.077948 15170 consensus_queue.cc:1035] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.078908 9210 raft_consensus.cc:2932] T b5780f4a85b146e2bfbbf7d2feaca316 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:58.079433 6922 raft_consensus.cc:1261] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.079542 12367 raft_consensus.cc:2932] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:58.079875 9081 raft_consensus.cc:2932] T fd45a0cf3a2d4657b02016bc60c5be34 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:58.081938 14177 raft_consensus.cc:1050] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43: attempting to promote NON_VOTER 5f46e19644fd4f4b825c86ba6af768da to VOTER
I20240503 13:59:58.082072 15170 consensus_queue.cc:1035] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.082378 14177 consensus_queue.cc:237] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 13:59:58.083534 9343 raft_consensus.cc:2932] T fd45a0cf3a2d4657b02016bc60c5be34 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:58.084365 6527 raft_consensus.cc:1261] T 63d2b0a8c3734720ba7d72a36d344cba P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEARNER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.084609 6387 catalog_manager.cc:5484] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:58.085305 14907 consensus_queue.cc:1035] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.085808 9081 raft_consensus.cc:1261] T 63d2b0a8c3734720ba7d72a36d344cba P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.086151 6791 raft_consensus.cc:2932] T fd45a0cf3a2d4657b02016bc60c5be34 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:58.086627 14907 consensus_queue.cc:1035] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.090003 9210 raft_consensus.cc:1261] T 63d2b0a8c3734720ba7d72a36d344cba P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.091223 14907 consensus_queue.cc:1035] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.093648 8946 consensus_queue.cc:237] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 13:59:58.094717 6791 raft_consensus.cc:1261] T fd45a0cf3a2d4657b02016bc60c5be34 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.094851 9081 raft_consensus.cc:1261] T fd45a0cf3a2d4657b02016bc60c5be34 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.094775 15169 raft_consensus.cc:2932] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:58.095199 14384 consensus_queue.cc:1035] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.095372 12367 consensus_queue.cc:1035] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.095798 15171 tablet_replica.cc:330] T b5780f4a85b146e2bfbbf7d2feaca316 P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:58.095695 14907 raft_consensus.cc:2932] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:58.095934 15171 raft_consensus.cc:2229] T b5780f4a85b146e2bfbbf7d2feaca316 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:58.096094 15171 raft_consensus.cc:2258] T b5780f4a85b146e2bfbbf7d2feaca316 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:58.096745 15171 ts_tablet_manager.cc:1875] T b5780f4a85b146e2bfbbf7d2feaca316 P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:58.097182 6922 raft_consensus.cc:2932] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:58.097206 14390 raft_consensus.cc:2932] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:58.097582 8946 raft_consensus.cc:2932] T 1e5875904e6047eb82f28c3cbc9318fc P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:58.099730 9081 raft_consensus.cc:2932] T 63d2b0a8c3734720ba7d72a36d344cba P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:58.099874 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fd45a0cf3a2d4657b02016bc60c5be34 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:58.100490 9080 raft_consensus.cc:2932] T fd45a0cf3a2d4657b02016bc60c5be34 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:58.100816 6791 raft_consensus.cc:2932] T fd45a0cf3a2d4657b02016bc60c5be34 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:58.100972 6526 raft_consensus.cc:2932] T 63d2b0a8c3734720ba7d72a36d344cba P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:58.102712 6384 catalog_manager.cc:5484] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:58.102751 6387 catalog_manager.cc:5484] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:58.106616 9210 raft_consensus.cc:2932] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:58.107213 15171 ts_tablet_manager.cc:1888] T b5780f4a85b146e2bfbbf7d2feaca316 P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:58.107304 15171 log.cc:1196] T b5780f4a85b146e2bfbbf7d2feaca316 P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/b5780f4a85b146e2bfbbf7d2feaca316
I20240503 13:59:58.110183 6387 catalog_manager.cc:5484] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index 3 to 4, VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:58.110994 9210 raft_consensus.cc:2932] T 63d2b0a8c3734720ba7d72a36d344cba P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:58.111330 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet b5780f4a85b146e2bfbbf7d2feaca316 (table rebalancer_extra_table_56 [id=bfd5a93b6dc141158909aa98a0ef7189]) successfully deleted
I20240503 13:59:58.114951 9321 tablet_service.cc:1514] Processing DeleteTablet for tablet fd45a0cf3a2d4657b02016bc60c5be34 with delete_type TABLET_DATA_TOMBSTONED (TS 0d77752181e944698e809e325e5937e6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41338
I20240503 13:59:58.115423 9343 consensus_queue.cc:237] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 13:59:58.117070 6658 consensus_queue.cc:237] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 13:59:58.118702 9210 raft_consensus.cc:1261] T 63d2b0a8c3734720ba7d72a36d344cba P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.119237 12915 consensus_queue.cc:1035] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.119448 6526 raft_consensus.cc:1261] T 63d2b0a8c3734720ba7d72a36d344cba P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.120270 12915 consensus_queue.cc:1035] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.120740 9210 raft_consensus.cc:1261] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.120785 6922 raft_consensus.cc:1261] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.121277 14432 consensus_queue.cc:1035] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.121414 12915 raft_consensus.cc:2932] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:58.121627 11378 consensus_queue.cc:1035] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.123046 6374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 63d2b0a8c3734720ba7d72a36d344cba with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:58.123107 9210 raft_consensus.cc:2932] T 63d2b0a8c3734720ba7d72a36d344cba P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:58.125178 6387 catalog_manager.cc:5484] T 63d2b0a8c3734720ba7d72a36d344cba P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:58.127876 15170 raft_consensus.cc:2932] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:58.129055 9205 raft_consensus.cc:2932] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:58.129312 9060 tablet_service.cc:1514] Processing DeleteTablet for tablet 63d2b0a8c3734720ba7d72a36d344cba with delete_type TABLET_DATA_TOMBSTONED (TS 8d574fb4a89d4eea947c859470d44dd4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:58.129580 15171 tablet_replica.cc:330] T 63d2b0a8c3734720ba7d72a36d344cba P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:58.129712 15171 raft_consensus.cc:2229] T 63d2b0a8c3734720ba7d72a36d344cba P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:58.129695 6374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1e5875904e6047eb82f28c3cbc9318fc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:58.129840 15171 raft_consensus.cc:2258] T 63d2b0a8c3734720ba7d72a36d344cba P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:58.130600 15171 ts_tablet_manager.cc:1875] T 63d2b0a8c3734720ba7d72a36d344cba P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:58.132125 6526 raft_consensus.cc:2932] T 63d2b0a8c3734720ba7d72a36d344cba P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:58.132867 15171 ts_tablet_manager.cc:1888] T 63d2b0a8c3734720ba7d72a36d344cba P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:58.132927 15171 log.cc:1196] T 63d2b0a8c3734720ba7d72a36d344cba P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/63d2b0a8c3734720ba7d72a36d344cba
I20240503 13:59:58.132948 6387 catalog_manager.cc:5484] T 1e5875904e6047eb82f28c3cbc9318fc P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:58.135052 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet 63d2b0a8c3734720ba7d72a36d344cba (table rebalancer_extra_table_32 [id=69fc27437d404cf4a64ea4f3de479bfc]) successfully deleted
I20240503 13:59:58.135861 8926 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e5875904e6047eb82f28c3cbc9318fc with delete_type TABLET_DATA_TOMBSTONED (TS 06ca8dbc004c42b99faa527a4e6f9545 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:58.136090 15143 tablet_replica.cc:330] T 1e5875904e6047eb82f28c3cbc9318fc P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:58.136250 15143 raft_consensus.cc:2229] T 1e5875904e6047eb82f28c3cbc9318fc P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:58.136416 15143 raft_consensus.cc:2258] T 1e5875904e6047eb82f28c3cbc9318fc P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:58.139624 15185 tablet_replica.cc:330] T fd45a0cf3a2d4657b02016bc60c5be34 P 0d77752181e944698e809e325e5937e6: stopping tablet replica
I20240503 13:59:58.139606 6922 raft_consensus.cc:2932] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:58.139817 15185 raft_consensus.cc:2229] T fd45a0cf3a2d4657b02016bc60c5be34 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:58.140020 15185 raft_consensus.cc:2258] T fd45a0cf3a2d4657b02016bc60c5be34 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:58.140750 15185 ts_tablet_manager.cc:1875] T fd45a0cf3a2d4657b02016bc60c5be34 P 0d77752181e944698e809e325e5937e6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:58.144712 15143 ts_tablet_manager.cc:1875] T 1e5875904e6047eb82f28c3cbc9318fc P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:58.146979 15143 ts_tablet_manager.cc:1888] T 1e5875904e6047eb82f28c3cbc9318fc P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:58.147048 15143 log.cc:1196] T 1e5875904e6047eb82f28c3cbc9318fc P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/1e5875904e6047eb82f28c3cbc9318fc
I20240503 13:59:58.147246 15185 ts_tablet_manager.cc:1888] T fd45a0cf3a2d4657b02016bc60c5be34 P 0d77752181e944698e809e325e5937e6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:58.147598 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 1e5875904e6047eb82f28c3cbc9318fc (table rebalancer_extra_table_2 [id=e92a6218e0734bbc97220157e127bc15]) successfully deleted
I20240503 13:59:58.147423 15185 log.cc:1196] T fd45a0cf3a2d4657b02016bc60c5be34 P 0d77752181e944698e809e325e5937e6: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/fd45a0cf3a2d4657b02016bc60c5be34
I20240503 13:59:58.148403 6374 catalog_manager.cc:4830] TS 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): tablet fd45a0cf3a2d4657b02016bc60c5be34 (table rebalancer_extra_table_32 [id=69fc27437d404cf4a64ea4f3de479bfc]) successfully deleted
I20240503 13:59:58.174999 9080 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "387d78763d574e57b914e2dbac83a932"
dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60802
I20240503 13:59:58.175185 9080 raft_consensus.cc:595] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:58.212807 6526 consensus_queue.cc:237] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:58.214151 6922 raft_consensus.cc:1261] T 342229593b184019ab915c0389b184b4 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.214411 6791 raft_consensus.cc:1261] T 342229593b184019ab915c0389b184b4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:59:58.214447 6465 consensus_peers.cc:485] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 342229593b184019ab915c0389b184b4. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:58.214632 15099 consensus_queue.cc:1035] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.214803 13157 consensus_queue.cc:1035] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.216715 15243 raft_consensus.cc:2932] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:58.217167 6922 raft_consensus.cc:2932] T 342229593b184019ab915c0389b184b4 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:58.217249 6791 raft_consensus.cc:2932] T 342229593b184019ab915c0389b184b4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:58.220685 6387 catalog_manager.cc:5484] T 342229593b184019ab915c0389b184b4 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:58.232012 6658 consensus_queue.cc:237] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 13:59:58.233650 6526 raft_consensus.cc:1261] T 7203f103a97141dcb70f42c0d27faa37 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:59:58.233804 6595 consensus_peers.cc:485] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 -> Peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): Couldn't send request to peer 06ca8dbc004c42b99faa527a4e6f9545. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7203f103a97141dcb70f42c0d27faa37. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:58.234066 6791 raft_consensus.cc:1261] T 7203f103a97141dcb70f42c0d27faa37 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.234046 15101 consensus_queue.cc:1035] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.234489 15101 consensus_queue.cc:1035] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.236651 14177 raft_consensus.cc:2932] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:58.237758 6791 raft_consensus.cc:2932] T 7203f103a97141dcb70f42c0d27faa37 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:58.240629 6387 catalog_manager.cc:5484] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:58.243592 6527 raft_consensus.cc:2932] T 7203f103a97141dcb70f42c0d27faa37 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:58.244542 6791 raft_consensus.cc:1203] T ab6fe502a80c410a9b6ee6f5d3d3205e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:58.257381 8946 consensus_queue.cc:237] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
W20240503 13:59:58.258996 8884 consensus_peers.cc:485] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): Couldn't send request to peer 2145d15ab2d74c0287839266f1411a43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ed7d05584d14fdaafb625eaff3ecf08. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:58.259061 6791 raft_consensus.cc:1261] T 2ed7d05584d14fdaafb625eaff3ecf08 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:58.259141 9079 raft_consensus.cc:1261] T 2ed7d05584d14fdaafb625eaff3ecf08 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:58.259594 12367 consensus_queue.cc:1035] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:58.259771 14390 consensus_queue.cc:1035] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:58.260208 15124 raft_consensus.cc:1050] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4: attempting to promote NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 to VOTER
I20240503 13:59:58.260632 15124 consensus_queue.cc:237] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
I20240503 13:59:58.261777 14390 raft_consensus.cc:2932] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:58.262048 8946 raft_consensus.cc:1261] T b8c8bc1c29934e32a3f30ad20db05462 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.262506 14146 consensus_queue.cc:1035] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.262580 6922 raft_consensus.cc:1261] T b8c8bc1c29934e32a3f30ad20db05462 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.262849 9205 raft_consensus.cc:1261] T b8c8bc1c29934e32a3f30ad20db05462 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.262997 6791 raft_consensus.cc:2932] T 2ed7d05584d14fdaafb625eaff3ecf08 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:58.263162 12372 consensus_queue.cc:1035] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.263833 13280 consensus_queue.cc:1035] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.265708 6384 catalog_manager.cc:5484] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:58.269354 13280 raft_consensus.cc:2932] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:58.270088 6922 raft_consensus.cc:2932] T b8c8bc1c29934e32a3f30ad20db05462 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:58.270314 9079 raft_consensus.cc:2932] T 2ed7d05584d14fdaafb625eaff3ecf08 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:58.272938 8946 raft_consensus.cc:2932] T b8c8bc1c29934e32a3f30ad20db05462 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:58.273125 6387 catalog_manager.cc:5484] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index 2 to 3, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:58.275271 9210 raft_consensus.cc:2932] T b8c8bc1c29934e32a3f30ad20db05462 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:58.277730 9079 consensus_queue.cc:237] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
I20240503 13:59:58.279139 6922 raft_consensus.cc:1261] T b8c8bc1c29934e32a3f30ad20db05462 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.279608 14146 consensus_queue.cc:1035] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.280573 8946 raft_consensus.cc:1261] T b8c8bc1c29934e32a3f30ad20db05462 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.280947 15124 consensus_queue.cc:1035] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.282647 15124 raft_consensus.cc:2932] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:58.284082 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b8c8bc1c29934e32a3f30ad20db05462 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:58.284202 8946 raft_consensus.cc:2932] T b8c8bc1c29934e32a3f30ad20db05462 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:58.284737 6658 consensus_queue.cc:237] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:58.286294 6791 raft_consensus.cc:1261] T 00434c9ea1654f7a9a9c31397a71998d P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.286530 6920 raft_consensus.cc:2932] T b8c8bc1c29934e32a3f30ad20db05462 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:58.286666 12915 consensus_queue.cc:1035] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.286854 6384 catalog_manager.cc:5484] T b8c8bc1c29934e32a3f30ad20db05462 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 13:59:58.287082 6596 consensus_peers.cc:485] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00434c9ea1654f7a9a9c31397a71998d. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:58.289700 9192 tablet_service.cc:1514] Processing DeleteTablet for tablet b8c8bc1c29934e32a3f30ad20db05462 with delete_type TABLET_DATA_TOMBSTONED (TS ab1d21935d6049888eb0ceccc3c9d735 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57960
I20240503 13:59:58.290094 6922 raft_consensus.cc:1261] T 00434c9ea1654f7a9a9c31397a71998d P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.290578 15101 consensus_queue.cc:1035] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.290762 14907 raft_consensus.cc:2932] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:58.291940 15273 tablet_replica.cc:330] T b8c8bc1c29934e32a3f30ad20db05462 P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 13:59:58.292135 15273 raft_consensus.cc:2229] T b8c8bc1c29934e32a3f30ad20db05462 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:58.292410 15273 raft_consensus.cc:2258] T b8c8bc1c29934e32a3f30ad20db05462 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:58.292371 6791 raft_consensus.cc:2932] T 00434c9ea1654f7a9a9c31397a71998d P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:58.293172 15273 ts_tablet_manager.cc:1875] T b8c8bc1c29934e32a3f30ad20db05462 P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:58.294667 6922 raft_consensus.cc:2932] T 00434c9ea1654f7a9a9c31397a71998d P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:58.296768 6384 catalog_manager.cc:5484] T 00434c9ea1654f7a9a9c31397a71998d P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:58.297179 15273 ts_tablet_manager.cc:1888] T b8c8bc1c29934e32a3f30ad20db05462 P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:58.297245 15273 log.cc:1196] T b8c8bc1c29934e32a3f30ad20db05462 P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/b8c8bc1c29934e32a3f30ad20db05462
I20240503 13:59:58.297991 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet b8c8bc1c29934e32a3f30ad20db05462 (table rebalancer_extra_table_26 [id=b9c4b58f1b704717873c5fbf98db4e2a]) successfully deleted
I20240503 13:59:58.306473 9079 consensus_queue.cc:237] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:58.308074 6791 raft_consensus.cc:1261] T 96db3ad20b864deb83dc3e2d64e2787e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:58.308176 9016 consensus_peers.cc:485] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 -> Peer ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): Couldn't send request to peer ab1d21935d6049888eb0ceccc3c9d735. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96db3ad20b864deb83dc3e2d64e2787e. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:58.308497 15124 consensus_queue.cc:1035] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:58.308622 6658 raft_consensus.cc:1261] T 96db3ad20b864deb83dc3e2d64e2787e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:58.308981 15124 consensus_queue.cc:1035] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:58.310223 12372 raft_consensus.cc:2932] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:58.311492 6791 raft_consensus.cc:2932] T 96db3ad20b864deb83dc3e2d64e2787e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:58.313812 6387 catalog_manager.cc:5484] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index -1 to 2, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:58.322093 6922 consensus_queue.cc:237] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 13:59:58.323499 6791 raft_consensus.cc:1261] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.323849 6659 raft_consensus.cc:1261] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.324028 15110 consensus_queue.cc:1035] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:59:58.324282 6857 consensus_peers.cc:485] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): Couldn't send request to peer 06ca8dbc004c42b99faa527a4e6f9545. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7fde7d6bb07e4dbda5276b0dbbcff19b. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:58.324323 12748 consensus_queue.cc:1035] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.326001 13370 raft_consensus.cc:2932] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:58.327471 6659 raft_consensus.cc:2932] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:58.329650 6379 catalog_manager.cc:5484] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:58.333137 6658 raft_consensus.cc:2932] T 96db3ad20b864deb83dc3e2d64e2787e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:58.339231 6791 raft_consensus.cc:2932] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:58.350488 9343 consensus_queue.cc:237] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
W20240503 13:59:58.351972 9278 consensus_peers.cc:485] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 -> Peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): Couldn't send request to peer f58065db70b64ef8b1dc499aa6ae4460. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1be8960eed8148f5958f6bb412c30a7e. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:58.352037 6791 raft_consensus.cc:1261] T 1be8960eed8148f5958f6bb412c30a7e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:58.352418 15170 consensus_queue.cc:1035] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:58.353050 9079 raft_consensus.cc:1261] T 1be8960eed8148f5958f6bb412c30a7e P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:58.353796 15169 consensus_queue.cc:1035] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:58.355811 11378 raft_consensus.cc:2932] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:58.356173 6791 raft_consensus.cc:2932] T 1be8960eed8148f5958f6bb412c30a7e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:58.356356 9080 raft_consensus.cc:2932] T 1be8960eed8148f5958f6bb412c30a7e P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:58.359567 6379 catalog_manager.cc:5484] T 1be8960eed8148f5958f6bb412c30a7e P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:58.422992 14880 raft_consensus.cc:1050] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64: attempting to promote NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 to VOTER
I20240503 13:59:58.423501 14880 consensus_queue.cc:237] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 13:59:58.425117 6526 raft_consensus.cc:1261] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:58.425774 9205 raft_consensus.cc:1261] T ce9410a4ed09491b8d37456ae44880a1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:58.425760 15149 consensus_queue.cc:1035] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:58.426223 15149 consensus_queue.cc:1035] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:58.427991 6659 raft_consensus.cc:1261] T ce9410a4ed09491b8d37456ae44880a1 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:58.429021 15149 raft_consensus.cc:2932] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:58.429275 14879 consensus_queue.cc:1035] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:58.429358 9205 raft_consensus.cc:2932] T ce9410a4ed09491b8d37456ae44880a1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:58.430840 12748 raft_consensus.cc:1050] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460: attempting to promote NON_VOTER 0d77752181e944698e809e325e5937e6 to VOTER
I20240503 13:59:58.431248 12748 consensus_queue.cc:237] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 13:59:58.431998 6526 raft_consensus.cc:2932] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:58.434230 6526 raft_consensus.cc:1261] T f403dc06195d449e8f454a9f4e795925 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.434715 15111 consensus_queue.cc:1035] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.434934 6658 raft_consensus.cc:1261] T f403dc06195d449e8f454a9f4e795925 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.435854 6387 catalog_manager.cc:5484] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:58.436940 13370 consensus_queue.cc:1035] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.438017 6659 raft_consensus.cc:2932] T ce9410a4ed09491b8d37456ae44880a1 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:58.439638 6791 consensus_queue.cc:237] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 13:59:58.441028 9205 raft_consensus.cc:1261] T ce9410a4ed09491b8d37456ae44880a1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:58.441622 6526 raft_consensus.cc:1261] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:58.442049 12032 consensus_queue.cc:1035] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:58.442076 12748 raft_consensus.cc:2932] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:58.442328 12032 consensus_queue.cc:1035] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:58.442689 6659 raft_consensus.cc:2932] T f403dc06195d449e8f454a9f4e795925 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:58.444751 9343 raft_consensus.cc:1261] T f403dc06195d449e8f454a9f4e795925 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.445528 6527 raft_consensus.cc:2932] T f403dc06195d449e8f454a9f4e795925 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:58.446535 15152 raft_consensus.cc:2932] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:58.447403 15110 consensus_queue.cc:1035] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.448202 6372 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ce9410a4ed09491b8d37456ae44880a1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:58.449105 9205 raft_consensus.cc:2932] T ce9410a4ed09491b8d37456ae44880a1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:58.449862 6387 catalog_manager.cc:5484] T f403dc06195d449e8f454a9f4e795925 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:58.451798 6384 catalog_manager.cc:5484] T ce9410a4ed09491b8d37456ae44880a1 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:58.453338 6526 raft_consensus.cc:2932] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:58.456910 12367 raft_consensus.cc:1050] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545: attempting to promote NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 to VOTER
I20240503 13:59:58.457307 12367 consensus_queue.cc:237] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 13:59:58.458750 9205 raft_consensus.cc:1261] T 038537d00e904bca89f0214d04fb4f13 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.458761 6922 raft_consensus.cc:1261] T 038537d00e904bca89f0214d04fb4f13 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.459182 14384 consensus_queue.cc:1035] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.459543 14384 consensus_queue.cc:1035] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.460088 9080 raft_consensus.cc:1261] T 038537d00e904bca89f0214d04fb4f13 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.460722 9343 raft_consensus.cc:2932] T f403dc06195d449e8f454a9f4e795925 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:58.460937 14384 consensus_queue.cc:1035] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.462910 6639 tablet_service.cc:1514] Processing DeleteTablet for tablet ce9410a4ed09491b8d37456ae44880a1 with delete_type TABLET_DATA_TOMBSTONED (TS 2145d15ab2d74c0287839266f1411a43 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:58.471123 14384 raft_consensus.cc:2932] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:58.471796 15371 tablet_replica.cc:330] T ce9410a4ed09491b8d37456ae44880a1 P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:58.471917 15371 raft_consensus.cc:2229] T ce9410a4ed09491b8d37456ae44880a1 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:58.472016 15371 raft_consensus.cc:2258] T ce9410a4ed09491b8d37456ae44880a1 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:58.473281 9080 raft_consensus.cc:2932] T 038537d00e904bca89f0214d04fb4f13 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:58.473572 9205 raft_consensus.cc:2932] T 038537d00e904bca89f0214d04fb4f13 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:58.474212 6380 catalog_manager.cc:5484] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:58.476635 15371 ts_tablet_manager.cc:1875] T ce9410a4ed09491b8d37456ae44880a1 P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:58.480365 12748 raft_consensus.cc:1050] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460: attempting to promote NON_VOTER 0d77752181e944698e809e325e5937e6 to VOTER
I20240503 13:59:58.480738 12748 consensus_queue.cc:237] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 13:59:58.482023 6788 raft_consensus.cc:1261] T af98fddcb623403c841a77ecc10a3e07 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:58.482093 6658 raft_consensus.cc:1261] T af98fddcb623403c841a77ecc10a3e07 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:58.482189 9342 raft_consensus.cc:1261] T af98fddcb623403c841a77ecc10a3e07 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:58.482336 15371 ts_tablet_manager.cc:1888] T ce9410a4ed09491b8d37456ae44880a1 P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:58.482414 15371 log.cc:1196] T ce9410a4ed09491b8d37456ae44880a1 P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/ce9410a4ed09491b8d37456ae44880a1
I20240503 13:59:58.482489 13370 consensus_queue.cc:1035] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:58.482647 15111 consensus_queue.cc:1035] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:58.483026 15111 consensus_queue.cc:1035] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:58.483888 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet ce9410a4ed09491b8d37456ae44880a1 (table rebalance_test_table_7 [id=73b39055500b4618821faa094d33670a]) successfully deleted
I20240503 13:59:58.486536 15111 raft_consensus.cc:2932] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:58.487344 6658 raft_consensus.cc:2932] T af98fddcb623403c841a77ecc10a3e07 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:58.488579 9342 raft_consensus.cc:2932] T af98fddcb623403c841a77ecc10a3e07 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:58.491855 6380 catalog_manager.cc:5484] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:58.494120 6922 raft_consensus.cc:2932] T 038537d00e904bca89f0214d04fb4f13 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:58.497465 6788 raft_consensus.cc:2932] T af98fddcb623403c841a77ecc10a3e07 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:58.497995 8947 consensus_queue.cc:237] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 13:59:58.499622 9205 raft_consensus.cc:1261] T 038537d00e904bca89f0214d04fb4f13 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.499956 6921 raft_consensus.cc:1261] T 038537d00e904bca89f0214d04fb4f13 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.500238 12367 consensus_queue.cc:1035] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.502328 13268 raft_consensus.cc:2932] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:58.502816 9205 raft_consensus.cc:2932] T 038537d00e904bca89f0214d04fb4f13 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:58.505441 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 038537d00e904bca89f0214d04fb4f13 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:58.505801 14390 consensus_queue.cc:1035] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.506880 6385 catalog_manager.cc:5484] T 038537d00e904bca89f0214d04fb4f13 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:58.507161 6921 raft_consensus.cc:2932] T 038537d00e904bca89f0214d04fb4f13 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:58.511693 9060 tablet_service.cc:1514] Processing DeleteTablet for tablet 038537d00e904bca89f0214d04fb4f13 with delete_type TABLET_DATA_TOMBSTONED (TS 8d574fb4a89d4eea947c859470d44dd4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:58.511906 15171 tablet_replica.cc:330] T 038537d00e904bca89f0214d04fb4f13 P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:58.512055 15171 raft_consensus.cc:2229] T 038537d00e904bca89f0214d04fb4f13 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:58.512198 15171 raft_consensus.cc:2258] T 038537d00e904bca89f0214d04fb4f13 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:58.512907 15171 ts_tablet_manager.cc:1875] T 038537d00e904bca89f0214d04fb4f13 P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:58.515195 15171 ts_tablet_manager.cc:1888] T 038537d00e904bca89f0214d04fb4f13 P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:58.515512 15171 log.cc:1196] T 038537d00e904bca89f0214d04fb4f13 P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/038537d00e904bca89f0214d04fb4f13
I20240503 13:59:58.516095 6922 consensus_queue.cc:237] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 13:59:58.516214 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet 038537d00e904bca89f0214d04fb4f13 (table rebalancer_extra_table_10 [id=b511123c500546da8d5ca21cb43ec824]) successfully deleted
I20240503 13:59:58.517414 9342 raft_consensus.cc:1261] T af98fddcb623403c841a77ecc10a3e07 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:58.518118 6788 raft_consensus.cc:1261] T af98fddcb623403c841a77ecc10a3e07 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:58.518517 15111 consensus_queue.cc:1035] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:58.518899 13370 consensus_queue.cc:1035] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:58.521646 13370 raft_consensus.cc:2932] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:58.522008 6788 raft_consensus.cc:2932] T af98fddcb623403c841a77ecc10a3e07 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:58.523715 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet af98fddcb623403c841a77ecc10a3e07 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:58.524078 9342 raft_consensus.cc:2932] T af98fddcb623403c841a77ecc10a3e07 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:58.528115 14146 raft_consensus.cc:979] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4: : Instructing follower 4a85e492c8154e9881a9cc2a105d7e64 to start an election
I20240503 13:59:58.528209 14146 raft_consensus.cc:1067] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Signalling peer 4a85e492c8154e9881a9cc2a105d7e64 to start an election
I20240503 13:59:58.528169 6380 catalog_manager.cc:5484] T af98fddcb623403c841a77ecc10a3e07 P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:58.529445 6788 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "387d78763d574e57b914e2dbac83a932"
dest_uuid: "4a85e492c8154e9881a9cc2a105d7e64"
from {username='slave'} at 127.0.99.134:52175
I20240503 13:59:58.529577 6788 raft_consensus.cc:482] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:58.529637 6788 raft_consensus.cc:3037] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:58.530797 6788 raft_consensus.cc:504] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 13:59:58.531225 6788 leader_election.cc:288] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 2 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:58.531394 6639 tablet_service.cc:1514] Processing DeleteTablet for tablet af98fddcb623403c841a77ecc10a3e07 with delete_type TABLET_DATA_TOMBSTONED (TS 2145d15ab2d74c0287839266f1411a43 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:58.531647 15371 tablet_replica.cc:330] T af98fddcb623403c841a77ecc10a3e07 P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:58.531785 15371 raft_consensus.cc:2229] T af98fddcb623403c841a77ecc10a3e07 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:58.531915 15371 raft_consensus.cc:2258] T af98fddcb623403c841a77ecc10a3e07 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:58.532222 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387d78763d574e57b914e2dbac83a932" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 13:59:58.532382 9080 raft_consensus.cc:3032] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:58.532366 6526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387d78763d574e57b914e2dbac83a932" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 13:59:58.532424 9080 raft_consensus.cc:729] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 13:59:58.532392 9210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387d78763d574e57b914e2dbac83a932" candidate_uuid: "4a85e492c8154e9881a9cc2a105d7e64" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 13:59:58.532486 6526 raft_consensus.cc:3037] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:58.532522 9210 raft_consensus.cc:3037] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:58.532606 9080 consensus_queue.cc:260] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 13:59:58.532819 9080 raft_consensus.cc:3037] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:58.532851 15371 ts_tablet_manager.cc:1875] T af98fddcb623403c841a77ecc10a3e07 P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:58.533546 9210 raft_consensus.cc:2454] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 2.
I20240503 13:59:58.533552 6526 raft_consensus.cc:2454] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 2.
I20240503 13:59:58.533993 9080 raft_consensus.cc:2454] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a85e492c8154e9881a9cc2a105d7e64 in term 2.
I20240503 13:59:58.535248 15371 ts_tablet_manager.cc:1888] T af98fddcb623403c841a77ecc10a3e07 P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:58.535336 15371 log.cc:1196] T af98fddcb623403c841a77ecc10a3e07 P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/af98fddcb623403c841a77ecc10a3e07
I20240503 13:59:58.535575 6726 leader_election.cc:302] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4a85e492c8154e9881a9cc2a105d7e64, 5f46e19644fd4f4b825c86ba6af768da, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 13:59:58.535902 15149 raft_consensus.cc:2781] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:58.535979 15149 raft_consensus.cc:686] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64 [term 2 LEADER]: Becoming Leader. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Running, Role: LEADER
I20240503 13:59:58.536124 15149 consensus_queue.cc:237] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 13:59:58.537065 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet af98fddcb623403c841a77ecc10a3e07 (table rebalance_test_table_1 [id=f757e59886674aafb5f975c30459df7e]) successfully deleted
I20240503 13:59:58.539971 6385 catalog_manager.cc:5484] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: term changed from 1 to 2, leader changed from 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) to 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:58.543607 15101 raft_consensus.cc:1050] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43: attempting to promote NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 to VOTER
I20240503 13:59:58.544061 15101 consensus_queue.cc:237] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
I20240503 13:59:58.545580 8946 raft_consensus.cc:1261] T efe57c7f306b4b0ab934388f5a372a74 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:58.545648 6921 raft_consensus.cc:1261] T efe57c7f306b4b0ab934388f5a372a74 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:58.545997 14907 consensus_queue.cc:1035] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:58.546139 12915 consensus_queue.cc:1035] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:58.546566 6788 raft_consensus.cc:1261] T efe57c7f306b4b0ab934388f5a372a74 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:58.546990 14907 consensus_queue.cc:1035] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:58.549057 14907 raft_consensus.cc:2932] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:58.550459 8947 raft_consensus.cc:2932] T efe57c7f306b4b0ab934388f5a372a74 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:58.552495 6921 raft_consensus.cc:2932] T efe57c7f306b4b0ab934388f5a372a74 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:58.553403 6387 catalog_manager.cc:5484] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:58.554657 6788 raft_consensus.cc:2932] T efe57c7f306b4b0ab934388f5a372a74 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:58.558226 6658 consensus_queue.cc:237] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
I20240503 13:59:58.560858 8947 raft_consensus.cc:1261] T efe57c7f306b4b0ab934388f5a372a74 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:58.561417 14907 consensus_queue.cc:1035] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:58.562165 6788 raft_consensus.cc:1261] T efe57c7f306b4b0ab934388f5a372a74 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:58.562503 14907 consensus_queue.cc:1035] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:58.572216 15111 raft_consensus.cc:1050] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460: attempting to promote NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 to VOTER
I20240503 13:59:58.572814 15111 consensus_queue.cc:237] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 13:59:58.574764 9210 raft_consensus.cc:1261] T 3331108688a3485a9c44c65600f0f3e5 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:58.574838 6526 raft_consensus.cc:1261] T 3331108688a3485a9c44c65600f0f3e5 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:58.575228 13370 consensus_queue.cc:1035] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:58.575639 15110 consensus_queue.cc:1035] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:58.577952 6527 consensus_queue.cc:237] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:58.579413 9210 raft_consensus.cc:1261] T 9a6709e4024d47448c20d33a2b36479e P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:58.579492 6659 raft_consensus.cc:1261] T 9a6709e4024d47448c20d33a2b36479e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:58.579861 15243 consensus_queue.cc:1035] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:58.581930 6791 raft_consensus.cc:1261] T 3331108688a3485a9c44c65600f0f3e5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:58.582000 15243 raft_consensus.cc:2932] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:58.582403 9210 raft_consensus.cc:2932] T 9a6709e4024d47448c20d33a2b36479e P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
W20240503 13:59:58.582810 6465 consensus_peers.cc:485] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a6709e4024d47448c20d33a2b36479e. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:58.583125 15099 consensus_queue.cc:1035] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:58.583214 13370 consensus_queue.cc:1035] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:58.585948 6380 catalog_manager.cc:5484] T 9a6709e4024d47448c20d33a2b36479e P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index -1 to 2, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:58.586015 6659 raft_consensus.cc:2932] T 9a6709e4024d47448c20d33a2b36479e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:58.590188 12915 raft_consensus.cc:2932] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:58.590880 6788 raft_consensus.cc:2932] T efe57c7f306b4b0ab934388f5a372a74 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:58.591310 15110 raft_consensus.cc:2932] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:58.591917 6526 raft_consensus.cc:2932] T 3331108688a3485a9c44c65600f0f3e5 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:58.594564 6387 catalog_manager.cc:5484] T efe57c7f306b4b0ab934388f5a372a74 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index 5 to 6, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:58.595167 6379 catalog_manager.cc:5484] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:58.595992 8946 raft_consensus.cc:2932] T efe57c7f306b4b0ab934388f5a372a74 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:58.595165 6374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet efe57c7f306b4b0ab934388f5a372a74 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:58.599673 6900 tablet_service.cc:1514] Processing DeleteTablet for tablet efe57c7f306b4b0ab934388f5a372a74 with delete_type TABLET_DATA_TOMBSTONED (TS f58065db70b64ef8b1dc499aa6ae4460 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:58.600121 15424 tablet_replica.cc:330] T efe57c7f306b4b0ab934388f5a372a74 P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:58.600452 15424 raft_consensus.cc:2229] T efe57c7f306b4b0ab934388f5a372a74 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:58.600760 15424 raft_consensus.cc:2258] T efe57c7f306b4b0ab934388f5a372a74 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:58.601593 15424 ts_tablet_manager.cc:1875] T efe57c7f306b4b0ab934388f5a372a74 P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:58.601625 9210 raft_consensus.cc:2932] T 3331108688a3485a9c44c65600f0f3e5 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:58.604425 15424 ts_tablet_manager.cc:1888] T efe57c7f306b4b0ab934388f5a372a74 P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:58.604921 15424 log.cc:1196] T efe57c7f306b4b0ab934388f5a372a74 P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/efe57c7f306b4b0ab934388f5a372a74
I20240503 13:59:58.605743 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet efe57c7f306b4b0ab934388f5a372a74 (table rebalance_test_table_15 [id=65cfc473fc21464c991e7a62f281b04e]) successfully deleted
I20240503 13:59:58.606104 6921 consensus_queue.cc:237] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 13:59:58.607391 9210 raft_consensus.cc:1261] T 3331108688a3485a9c44c65600f0f3e5 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:58.608064 6526 raft_consensus.cc:1261] T 3331108688a3485a9c44c65600f0f3e5 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:58.608139 12748 consensus_queue.cc:1035] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:58.608645 12748 consensus_queue.cc:1035] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:58.611084 12748 raft_consensus.cc:2932] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:58.612650 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3331108688a3485a9c44c65600f0f3e5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:58.612848 6526 raft_consensus.cc:2932] T 3331108688a3485a9c44c65600f0f3e5 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:58.613620 6658 consensus_queue.cc:237] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 13:59:58.615717 6788 raft_consensus.cc:1261] T 9de4c2ac66344f719b3fe56ffbcc804d P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.615953 6380 catalog_manager.cc:5484] T 3331108688a3485a9c44c65600f0f3e5 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 5 to 6, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:58.615996 9210 raft_consensus.cc:2932] T 3331108688a3485a9c44c65600f0f3e5 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:58.617532 14907 consensus_queue.cc:1035] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:59:58.617794 6595 consensus_peers.cc:485] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 -> Peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): Couldn't send request to peer 06ca8dbc004c42b99faa527a4e6f9545. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9de4c2ac66344f719b3fe56ffbcc804d. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:58.618016 6526 raft_consensus.cc:1261] T 9de4c2ac66344f719b3fe56ffbcc804d P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.619314 14907 consensus_queue.cc:1035] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.619338 6769 tablet_service.cc:1514] Processing DeleteTablet for tablet 3331108688a3485a9c44c65600f0f3e5 with delete_type TABLET_DATA_TOMBSTONED (TS 4a85e492c8154e9881a9cc2a105d7e64 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:58.620571 15427 tablet_replica.cc:330] T 3331108688a3485a9c44c65600f0f3e5 P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:58.620725 15427 raft_consensus.cc:2229] T 3331108688a3485a9c44c65600f0f3e5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:58.620868 15427 pending_rounds.cc:62] T 3331108688a3485a9c44c65600f0f3e5 P 4a85e492c8154e9881a9cc2a105d7e64: Trying to abort 1 pending ops.
I20240503 13:59:58.620936 15427 pending_rounds.cc:68] T 3331108688a3485a9c44c65600f0f3e5 P 4a85e492c8154e9881a9cc2a105d7e64: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023594694952763392 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "3331108688a3485a9c44c65600f0f3e5" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } } }
I20240503 13:59:58.621094 15427 raft_consensus.cc:2894] T 3331108688a3485a9c44c65600f0f3e5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 13:59:58.621155 15427 raft_consensus.cc:2866] T 3331108688a3485a9c44c65600f0f3e5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 13:59:58.621201 15427 raft_consensus.cc:2258] T 3331108688a3485a9c44c65600f0f3e5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 13:59:58.622906 15418 consensus_queue.cc:1159] T 3331108688a3485a9c44c65600f0f3e5 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4a85e492c8154e9881a9cc2a105d7e64" status { last_received { term: 1 index: 5 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 5 } }
I20240503 13:59:58.623324 14146 raft_consensus.cc:1050] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4: attempting to promote NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 to VOTER
I20240503 13:59:58.623725 14146 consensus_queue.cc:237] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 13:59:58.623804 15427 ts_tablet_manager.cc:1875] T 3331108688a3485a9c44c65600f0f3e5 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:58.625041 9342 raft_consensus.cc:1261] T ab6fe502a80c410a9b6ee6f5d3d3205e P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.625041 8946 raft_consensus.cc:1261] T ab6fe502a80c410a9b6ee6f5d3d3205e P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.625406 13280 consensus_queue.cc:1035] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.625543 12372 consensus_queue.cc:1035] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.625793 6526 raft_consensus.cc:2932] T 9de4c2ac66344f719b3fe56ffbcc804d P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:58.626197 6790 raft_consensus.cc:1261] T ab6fe502a80c410a9b6ee6f5d3d3205e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.626507 14907 raft_consensus.cc:2932] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:58.627435 13280 consensus_queue.cc:1035] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.630322 15124 raft_consensus.cc:2932] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:58.630137 6387 catalog_manager.cc:5484] T 9de4c2ac66344f719b3fe56ffbcc804d P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:58.631752 6790 raft_consensus.cc:2932] T 9de4c2ac66344f719b3fe56ffbcc804d P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:58.632050 8946 raft_consensus.cc:2932] T ab6fe502a80c410a9b6ee6f5d3d3205e P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:58.633828 15427 ts_tablet_manager.cc:1888] T 3331108688a3485a9c44c65600f0f3e5 P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:58.633962 15427 log.cc:1196] T 3331108688a3485a9c44c65600f0f3e5 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/3331108688a3485a9c44c65600f0f3e5
I20240503 13:59:58.635509 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet 3331108688a3485a9c44c65600f0f3e5 (table rebalance_test_table_3 [id=0ea567ca26f349a788891ba7d2d3751a]) successfully deleted
I20240503 13:59:58.636322 12367 raft_consensus.cc:1050] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545: attempting to promote NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 to VOTER
I20240503 13:59:58.636848 12367 consensus_queue.cc:237] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 13:59:58.638222 9080 raft_consensus.cc:1261] T 174e5b7577d94d87b729d1735987a97a P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.638504 9210 raft_consensus.cc:1261] T 174e5b7577d94d87b729d1735987a97a P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.639075 14384 consensus_queue.cc:1035] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.639087 6379 catalog_manager.cc:5484] T ab6fe502a80c410a9b6ee6f5d3d3205e P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:58.639446 14384 consensus_queue.cc:1035] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.639484 9343 raft_consensus.cc:1261] T 174e5b7577d94d87b729d1735987a97a P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:58.640403 14390 consensus_queue.cc:1035] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:58.643301 14384 raft_consensus.cc:2932] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:58.643478 6790 raft_consensus.cc:2932] T ab6fe502a80c410a9b6ee6f5d3d3205e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:58.643783 9343 raft_consensus.cc:2932] T 174e5b7577d94d87b729d1735987a97a P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:58.645170 9080 raft_consensus.cc:2932] T 174e5b7577d94d87b729d1735987a97a P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:58.646996 9210 raft_consensus.cc:2932] T 174e5b7577d94d87b729d1735987a97a P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:58.649991 6380 catalog_manager.cc:5484] T 174e5b7577d94d87b729d1735987a97a P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index 2 to 3, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:58.654029 8946 consensus_queue.cc:237] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 13:59:58.655635 9080 raft_consensus.cc:1261] T 174e5b7577d94d87b729d1735987a97a P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.656342 12367 consensus_queue.cc:1035] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.656497 15434 ts_tablet_manager.cc:902] T 7203f103a97141dcb70f42c0d27faa37 P 06ca8dbc004c42b99faa527a4e6f9545: Initiating tablet copy from peer 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:58.657078 15434 tablet_copy_client.cc:308] T 7203f103a97141dcb70f42c0d27faa37 P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46119
I20240503 13:59:58.657640 15444 ts_tablet_manager.cc:902] T 342229593b184019ab915c0389b184b4 P 0d77752181e944698e809e325e5937e6: Initiating tablet copy from peer 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:58.658001 6680 tablet_copy_service.cc:140] P 2145d15ab2d74c0287839266f1411a43: Received BeginTabletCopySession request for tablet 7203f103a97141dcb70f42c0d27faa37 from peer 06ca8dbc004c42b99faa527a4e6f9545 ({username='slave'} at 127.0.99.133:36361)
I20240503 13:59:58.658110 6680 tablet_copy_service.cc:161] P 2145d15ab2d74c0287839266f1411a43: Beginning new tablet copy session on tablet 7203f103a97141dcb70f42c0d27faa37 from peer 06ca8dbc004c42b99faa527a4e6f9545 at {username='slave'} at 127.0.99.133:36361: session id = 06ca8dbc004c42b99faa527a4e6f9545-7203f103a97141dcb70f42c0d27faa37
I20240503 13:59:58.659212 6680 tablet_copy_source_session.cc:205] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:58.660151 14384 raft_consensus.cc:2932] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:58.660578 9080 raft_consensus.cc:2932] T 174e5b7577d94d87b729d1735987a97a P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:58.661193 9342 raft_consensus.cc:2932] T ab6fe502a80c410a9b6ee6f5d3d3205e P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:58.662322 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 174e5b7577d94d87b729d1735987a97a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:58.663089 15444 tablet_copy_client.cc:308] T 342229593b184019ab915c0389b184b4 P 0d77752181e944698e809e325e5937e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38021
I20240503 13:59:58.663664 6549 tablet_copy_service.cc:140] P 5f46e19644fd4f4b825c86ba6af768da: Received BeginTabletCopySession request for tablet 342229593b184019ab915c0389b184b4 from peer 0d77752181e944698e809e325e5937e6 ({username='slave'} at 127.0.99.136:52973)
I20240503 13:59:58.663794 6549 tablet_copy_service.cc:161] P 5f46e19644fd4f4b825c86ba6af768da: Beginning new tablet copy session on tablet 342229593b184019ab915c0389b184b4 from peer 0d77752181e944698e809e325e5937e6 at {username='slave'} at 127.0.99.136:52973: session id = 0d77752181e944698e809e325e5937e6-342229593b184019ab915c0389b184b4
I20240503 13:59:58.664389 9342 raft_consensus.cc:1261] T 174e5b7577d94d87b729d1735987a97a P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.664784 6549 tablet_copy_source_session.cc:205] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:58.664737 9080 consensus_queue.cc:237] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 13:59:58.661484 15434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7203f103a97141dcb70f42c0d27faa37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:58.665343 13268 consensus_queue.cc:1035] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.665861 6788 raft_consensus.cc:1261] T ab6fe502a80c410a9b6ee6f5d3d3205e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.666045 8947 consensus_queue.cc:237] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:58.666229 12372 consensus_queue.cc:1035] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.666838 9342 raft_consensus.cc:2932] T 174e5b7577d94d87b729d1735987a97a P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:58.667507 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 342229593b184019ab915c0389b184b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:58.667598 9343 raft_consensus.cc:1261] T ab6fe502a80c410a9b6ee6f5d3d3205e P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.668109 12372 consensus_queue.cc:1035] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:59:58.670214 8884 consensus_peers.cc:485] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): Couldn't send request to peer 2145d15ab2d74c0287839266f1411a43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96d1a236cdb7406090021842d8dbc5a0. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:58.670305 6527 raft_consensus.cc:1261] T 96d1a236cdb7406090021842d8dbc5a0 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:58.671099 12367 consensus_queue.cc:1035] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:58.671675 12372 raft_consensus.cc:2932] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:58.668823 6380 catalog_manager.cc:5484] T 174e5b7577d94d87b729d1735987a97a P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 13:59:58.672710 9343 raft_consensus.cc:2932] T ab6fe502a80c410a9b6ee6f5d3d3205e P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:58.673863 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ab6fe502a80c410a9b6ee6f5d3d3205e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:58.674331 15444 tablet_copy_client.cc:783] T 342229593b184019ab915c0389b184b4 P 0d77752181e944698e809e325e5937e6: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:58.674500 15444 tablet_copy_client.cc:647] T 342229593b184019ab915c0389b184b4 P 0d77752181e944698e809e325e5937e6: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:58.674670 6790 raft_consensus.cc:2932] T ab6fe502a80c410a9b6ee6f5d3d3205e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:58.674374 9080 raft_consensus.cc:1261] T 96d1a236cdb7406090021842d8dbc5a0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:58.675408 12367 consensus_queue.cc:1035] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:58.676491 9192 tablet_service.cc:1514] Processing DeleteTablet for tablet 174e5b7577d94d87b729d1735987a97a with delete_type TABLET_DATA_TOMBSTONED (TS ab1d21935d6049888eb0ceccc3c9d735 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57960
I20240503 13:59:58.676776 15273 tablet_replica.cc:330] T 174e5b7577d94d87b729d1735987a97a P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 13:59:58.676944 15273 raft_consensus.cc:2229] T 174e5b7577d94d87b729d1735987a97a P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:58.677089 15273 raft_consensus.cc:2258] T 174e5b7577d94d87b729d1735987a97a P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:58.677345 15434 tablet_copy_client.cc:783] T 7203f103a97141dcb70f42c0d27faa37 P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:58.677629 15434 tablet_copy_client.cc:647] T 7203f103a97141dcb70f42c0d27faa37 P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:58.678232 6385 catalog_manager.cc:5484] T ab6fe502a80c410a9b6ee6f5d3d3205e P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 13:59:58.678411 15273 ts_tablet_manager.cc:1875] T 174e5b7577d94d87b729d1735987a97a P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:58.680087 15434 tablet_copy_client.cc:523] T 7203f103a97141dcb70f42c0d27faa37 P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:58.681365 15273 ts_tablet_manager.cc:1888] T 174e5b7577d94d87b729d1735987a97a P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:58.681429 15273 log.cc:1196] T 174e5b7577d94d87b729d1735987a97a P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/174e5b7577d94d87b729d1735987a97a
I20240503 13:59:58.681450 15434 tablet_bootstrap.cc:492] T 7203f103a97141dcb70f42c0d27faa37 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:58.682018 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet 174e5b7577d94d87b729d1735987a97a (table rebalancer_extra_table_34 [id=22ad55d53adb4db5b7dd551f6384d11b]) successfully deleted
I20240503 13:59:58.683933 13268 raft_consensus.cc:2932] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:58.685312 8926 tablet_service.cc:1514] Processing DeleteTablet for tablet ab6fe502a80c410a9b6ee6f5d3d3205e with delete_type TABLET_DATA_TOMBSTONED (TS 06ca8dbc004c42b99faa527a4e6f9545 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38372
I20240503 13:59:58.685590 15457 tablet_replica.cc:330] T ab6fe502a80c410a9b6ee6f5d3d3205e P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 13:59:58.685994 6526 raft_consensus.cc:2932] T 96d1a236cdb7406090021842d8dbc5a0 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:58.689021 15457 raft_consensus.cc:2229] T ab6fe502a80c410a9b6ee6f5d3d3205e P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:58.689198 15457 raft_consensus.cc:2258] T ab6fe502a80c410a9b6ee6f5d3d3205e P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:58.691210 6380 catalog_manager.cc:5484] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:58.691720 15444 tablet_copy_client.cc:523] T 342229593b184019ab915c0389b184b4 P 0d77752181e944698e809e325e5937e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:58.693413 15444 tablet_bootstrap.cc:492] T 342229593b184019ab915c0389b184b4 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:58.693720 15457 ts_tablet_manager.cc:1875] T ab6fe502a80c410a9b6ee6f5d3d3205e P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:58.696631 15457 ts_tablet_manager.cc:1888] T ab6fe502a80c410a9b6ee6f5d3d3205e P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:58.696698 15457 log.cc:1196] T ab6fe502a80c410a9b6ee6f5d3d3205e P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/ab6fe502a80c410a9b6ee6f5d3d3205e
I20240503 13:59:58.697237 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet ab6fe502a80c410a9b6ee6f5d3d3205e (table rebalancer_extra_table_4 [id=7f5a05b637884ebab6fb86d880c343e4]) successfully deleted
I20240503 13:59:58.709030 9080 raft_consensus.cc:2932] T 96d1a236cdb7406090021842d8dbc5a0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 13:59:58.716352 15434 tablet_bootstrap.cc:492] T 7203f103a97141dcb70f42c0d27faa37 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:58.716688 15434 tablet_bootstrap.cc:492] T 7203f103a97141dcb70f42c0d27faa37 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap complete.
I20240503 13:59:58.717003 15434 ts_tablet_manager.cc:1367] T 7203f103a97141dcb70f42c0d27faa37 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.036s user 0.027s sys 0.000s
I20240503 13:59:58.717658 15434 raft_consensus.cc:348] T 7203f103a97141dcb70f42c0d27faa37 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 13:59:58.717826 15434 raft_consensus.cc:729] T 7203f103a97141dcb70f42c0d27faa37 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: LEARNER
I20240503 13:59:58.717939 15434 consensus_queue.cc:260] T 7203f103a97141dcb70f42c0d27faa37 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 13:59:58.718338 15434 ts_tablet_manager.cc:1398] T 7203f103a97141dcb70f42c0d27faa37 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:58.719974 6680 tablet_copy_service.cc:342] P 2145d15ab2d74c0287839266f1411a43: Request end of tablet copy session 06ca8dbc004c42b99faa527a4e6f9545-7203f103a97141dcb70f42c0d27faa37 received from {username='slave'} at 127.0.99.133:36361
I20240503 13:59:58.720059 6680 tablet_copy_service.cc:433] P 2145d15ab2d74c0287839266f1411a43: ending tablet copy session 06ca8dbc004c42b99faa527a4e6f9545-7203f103a97141dcb70f42c0d27faa37 on tablet 7203f103a97141dcb70f42c0d27faa37 with peer 06ca8dbc004c42b99faa527a4e6f9545
I20240503 13:59:58.731231 15444 tablet_bootstrap.cc:492] T 342229593b184019ab915c0389b184b4 P 0d77752181e944698e809e325e5937e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:58.739056 15444 tablet_bootstrap.cc:492] T 342229593b184019ab915c0389b184b4 P 0d77752181e944698e809e325e5937e6: Bootstrap complete.
I20240503 13:59:58.739315 15444 ts_tablet_manager.cc:1367] T 342229593b184019ab915c0389b184b4 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.046s user 0.022s sys 0.003s
I20240503 13:59:58.740101 15444 raft_consensus.cc:348] T 342229593b184019ab915c0389b184b4 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:58.751412 15444 raft_consensus.cc:729] T 342229593b184019ab915c0389b184b4 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: LEARNER
I20240503 13:59:58.751540 15444 consensus_queue.cc:260] T 342229593b184019ab915c0389b184b4 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:58.752877 15444 ts_tablet_manager.cc:1398] T 342229593b184019ab915c0389b184b4 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240503 13:59:58.753583 6549 tablet_copy_service.cc:342] P 5f46e19644fd4f4b825c86ba6af768da: Request end of tablet copy session 0d77752181e944698e809e325e5937e6-342229593b184019ab915c0389b184b4 received from {username='slave'} at 127.0.99.136:52973
I20240503 13:59:58.753660 6549 tablet_copy_service.cc:433] P 5f46e19644fd4f4b825c86ba6af768da: ending tablet copy session 0d77752181e944698e809e325e5937e6-342229593b184019ab915c0389b184b4 on tablet 342229593b184019ab915c0389b184b4 with peer 0d77752181e944698e809e325e5937e6
I20240503 13:59:58.782423 15444 ts_tablet_manager.cc:902] T 00434c9ea1654f7a9a9c31397a71998d P 0d77752181e944698e809e325e5937e6: Initiating tablet copy from peer 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:58.782935 15444 tablet_copy_client.cc:308] T 00434c9ea1654f7a9a9c31397a71998d P 0d77752181e944698e809e325e5937e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46119
I20240503 13:59:58.783463 6680 tablet_copy_service.cc:140] P 2145d15ab2d74c0287839266f1411a43: Received BeginTabletCopySession request for tablet 00434c9ea1654f7a9a9c31397a71998d from peer 0d77752181e944698e809e325e5937e6 ({username='slave'} at 127.0.99.136:46439)
I20240503 13:59:58.783577 6680 tablet_copy_service.cc:161] P 2145d15ab2d74c0287839266f1411a43: Beginning new tablet copy session on tablet 00434c9ea1654f7a9a9c31397a71998d from peer 0d77752181e944698e809e325e5937e6 at {username='slave'} at 127.0.99.136:46439: session id = 0d77752181e944698e809e325e5937e6-00434c9ea1654f7a9a9c31397a71998d
I20240503 13:59:58.784655 6680 tablet_copy_source_session.cc:205] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:58.785817 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00434c9ea1654f7a9a9c31397a71998d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:58.788537 15508 ts_tablet_manager.cc:902] T 2ed7d05584d14fdaafb625eaff3ecf08 P 2145d15ab2d74c0287839266f1411a43: Initiating tablet copy from peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:58.789405 15444 tablet_copy_client.cc:783] T 00434c9ea1654f7a9a9c31397a71998d P 0d77752181e944698e809e325e5937e6: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:58.790472 15444 tablet_copy_client.cc:647] T 00434c9ea1654f7a9a9c31397a71998d P 0d77752181e944698e809e325e5937e6: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:58.790963 15508 tablet_copy_client.cc:308] T 2ed7d05584d14fdaafb625eaff3ecf08 P 2145d15ab2d74c0287839266f1411a43: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:34547
I20240503 13:59:58.791831 15444 tablet_copy_client.cc:523] T 00434c9ea1654f7a9a9c31397a71998d P 0d77752181e944698e809e325e5937e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:58.791878 8968 tablet_copy_service.cc:140] P 06ca8dbc004c42b99faa527a4e6f9545: Received BeginTabletCopySession request for tablet 2ed7d05584d14fdaafb625eaff3ecf08 from peer 2145d15ab2d74c0287839266f1411a43 ({username='slave'} at 127.0.99.130:39847)
I20240503 13:59:58.791994 8968 tablet_copy_service.cc:161] P 06ca8dbc004c42b99faa527a4e6f9545: Beginning new tablet copy session on tablet 2ed7d05584d14fdaafb625eaff3ecf08 from peer 2145d15ab2d74c0287839266f1411a43 at {username='slave'} at 127.0.99.130:39847: session id = 2145d15ab2d74c0287839266f1411a43-2ed7d05584d14fdaafb625eaff3ecf08
I20240503 13:59:58.792831 8968 tablet_copy_source_session.cc:205] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:58.793097 15444 tablet_bootstrap.cc:492] T 00434c9ea1654f7a9a9c31397a71998d P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:58.794138 15508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed7d05584d14fdaafb625eaff3ecf08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:58.796558 15508 tablet_copy_client.cc:783] T 2ed7d05584d14fdaafb625eaff3ecf08 P 2145d15ab2d74c0287839266f1411a43: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:58.796716 15508 tablet_copy_client.cc:647] T 2ed7d05584d14fdaafb625eaff3ecf08 P 2145d15ab2d74c0287839266f1411a43: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:58.798532 15508 tablet_copy_client.cc:523] T 2ed7d05584d14fdaafb625eaff3ecf08 P 2145d15ab2d74c0287839266f1411a43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:58.799942 15508 tablet_bootstrap.cc:492] T 2ed7d05584d14fdaafb625eaff3ecf08 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:58.804500 15508 tablet_bootstrap.cc:492] T 2ed7d05584d14fdaafb625eaff3ecf08 P 2145d15ab2d74c0287839266f1411a43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:58.804836 15508 tablet_bootstrap.cc:492] T 2ed7d05584d14fdaafb625eaff3ecf08 P 2145d15ab2d74c0287839266f1411a43: Bootstrap complete.
I20240503 13:59:58.804998 15508 ts_tablet_manager.cc:1367] T 2ed7d05584d14fdaafb625eaff3ecf08 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 13:59:58.805420 15508 raft_consensus.cc:348] T 2ed7d05584d14fdaafb625eaff3ecf08 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:58.805564 15508 raft_consensus.cc:729] T 2ed7d05584d14fdaafb625eaff3ecf08 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: LEARNER
I20240503 13:59:58.805670 15508 consensus_queue.cc:260] T 2ed7d05584d14fdaafb625eaff3ecf08 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:58.806008 15508 ts_tablet_manager.cc:1398] T 2ed7d05584d14fdaafb625eaff3ecf08 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:58.808779 8968 tablet_copy_service.cc:342] P 06ca8dbc004c42b99faa527a4e6f9545: Request end of tablet copy session 2145d15ab2d74c0287839266f1411a43-2ed7d05584d14fdaafb625eaff3ecf08 received from {username='slave'} at 127.0.99.130:39847
I20240503 13:59:58.808869 8968 tablet_copy_service.cc:433] P 06ca8dbc004c42b99faa527a4e6f9545: ending tablet copy session 2145d15ab2d74c0287839266f1411a43-2ed7d05584d14fdaafb625eaff3ecf08 on tablet 2ed7d05584d14fdaafb625eaff3ecf08 with peer 2145d15ab2d74c0287839266f1411a43
I20240503 13:59:58.813071 15434 ts_tablet_manager.cc:902] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 06ca8dbc004c42b99faa527a4e6f9545: Initiating tablet copy from peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:58.813510 15434 tablet_copy_client.cc:308] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43531
I20240503 13:59:58.814220 6942 tablet_copy_service.cc:140] P f58065db70b64ef8b1dc499aa6ae4460: Received BeginTabletCopySession request for tablet 7fde7d6bb07e4dbda5276b0dbbcff19b from peer 06ca8dbc004c42b99faa527a4e6f9545 ({username='slave'} at 127.0.99.133:60451)
I20240503 13:59:58.814342 6942 tablet_copy_service.cc:161] P f58065db70b64ef8b1dc499aa6ae4460: Beginning new tablet copy session on tablet 7fde7d6bb07e4dbda5276b0dbbcff19b from peer 06ca8dbc004c42b99faa527a4e6f9545 at {username='slave'} at 127.0.99.133:60451: session id = 06ca8dbc004c42b99faa527a4e6f9545-7fde7d6bb07e4dbda5276b0dbbcff19b
I20240503 13:59:58.815441 6942 tablet_copy_source_session.cc:205] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:58.818729 15434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fde7d6bb07e4dbda5276b0dbbcff19b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:58.822906 15434 tablet_copy_client.cc:783] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:58.824333 15434 tablet_copy_client.cc:647] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:58.825439 15444 tablet_bootstrap.cc:492] T 00434c9ea1654f7a9a9c31397a71998d P 0d77752181e944698e809e325e5937e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:58.825583 15434 tablet_copy_client.cc:523] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:58.825731 15444 tablet_bootstrap.cc:492] T 00434c9ea1654f7a9a9c31397a71998d P 0d77752181e944698e809e325e5937e6: Bootstrap complete.
I20240503 13:59:58.825964 15444 ts_tablet_manager.cc:1367] T 00434c9ea1654f7a9a9c31397a71998d P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.033s user 0.028s sys 0.002s
I20240503 13:59:58.826452 15444 raft_consensus.cc:348] T 00434c9ea1654f7a9a9c31397a71998d P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:58.826575 15444 raft_consensus.cc:729] T 00434c9ea1654f7a9a9c31397a71998d P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: LEARNER
I20240503 13:59:58.826678 15444 consensus_queue.cc:260] T 00434c9ea1654f7a9a9c31397a71998d P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:58.826970 15444 ts_tablet_manager.cc:1398] T 00434c9ea1654f7a9a9c31397a71998d P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:58.827966 15434 tablet_bootstrap.cc:492] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:58.828112 6680 tablet_copy_service.cc:342] P 2145d15ab2d74c0287839266f1411a43: Request end of tablet copy session 0d77752181e944698e809e325e5937e6-00434c9ea1654f7a9a9c31397a71998d received from {username='slave'} at 127.0.99.136:46439
I20240503 13:59:58.828234 6680 tablet_copy_service.cc:433] P 2145d15ab2d74c0287839266f1411a43: ending tablet copy session 0d77752181e944698e809e325e5937e6-00434c9ea1654f7a9a9c31397a71998d on tablet 00434c9ea1654f7a9a9c31397a71998d with peer 0d77752181e944698e809e325e5937e6
I20240503 13:59:58.839581 15535 ts_tablet_manager.cc:902] T 96db3ad20b864deb83dc3e2d64e2787e P ab1d21935d6049888eb0ceccc3c9d735: Initiating tablet copy from peer 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 13:59:58.840698 15535 tablet_copy_client.cc:308] T 96db3ad20b864deb83dc3e2d64e2787e P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:39481
I20240503 13:59:58.841120 9101 tablet_copy_service.cc:140] P 8d574fb4a89d4eea947c859470d44dd4: Received BeginTabletCopySession request for tablet 96db3ad20b864deb83dc3e2d64e2787e from peer ab1d21935d6049888eb0ceccc3c9d735 ({username='slave'} at 127.0.99.135:32801)
I20240503 13:59:58.841226 9101 tablet_copy_service.cc:161] P 8d574fb4a89d4eea947c859470d44dd4: Beginning new tablet copy session on tablet 96db3ad20b864deb83dc3e2d64e2787e from peer ab1d21935d6049888eb0ceccc3c9d735 at {username='slave'} at 127.0.99.135:32801: session id = ab1d21935d6049888eb0ceccc3c9d735-96db3ad20b864deb83dc3e2d64e2787e
I20240503 13:59:58.842161 9101 tablet_copy_source_session.cc:205] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:58.843108 15535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96db3ad20b864deb83dc3e2d64e2787e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:58.846197 15535 tablet_copy_client.cc:783] T 96db3ad20b864deb83dc3e2d64e2787e P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:58.846514 15535 tablet_copy_client.cc:647] T 96db3ad20b864deb83dc3e2d64e2787e P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:58.847755 15535 tablet_copy_client.cc:523] T 96db3ad20b864deb83dc3e2d64e2787e P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:58.849941 15535 tablet_bootstrap.cc:492] T 96db3ad20b864deb83dc3e2d64e2787e P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:58.854657 15535 tablet_bootstrap.cc:492] T 96db3ad20b864deb83dc3e2d64e2787e P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:58.854920 15535 tablet_bootstrap.cc:492] T 96db3ad20b864deb83dc3e2d64e2787e P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap complete.
I20240503 13:59:58.855080 15535 ts_tablet_manager.cc:1367] T 96db3ad20b864deb83dc3e2d64e2787e P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240503 13:59:58.855556 15535 raft_consensus.cc:348] T 96db3ad20b864deb83dc3e2d64e2787e P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:58.855676 15535 raft_consensus.cc:729] T 96db3ad20b864deb83dc3e2d64e2787e P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: LEARNER
I20240503 13:59:58.855768 15535 consensus_queue.cc:260] T 96db3ad20b864deb83dc3e2d64e2787e P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:58.856380 15535 ts_tablet_manager.cc:1398] T 96db3ad20b864deb83dc3e2d64e2787e P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 13:59:58.857162 9101 tablet_copy_service.cc:342] P 8d574fb4a89d4eea947c859470d44dd4: Request end of tablet copy session ab1d21935d6049888eb0ceccc3c9d735-96db3ad20b864deb83dc3e2d64e2787e received from {username='slave'} at 127.0.99.135:32801
I20240503 13:59:58.857259 9101 tablet_copy_service.cc:433] P 8d574fb4a89d4eea947c859470d44dd4: ending tablet copy session ab1d21935d6049888eb0ceccc3c9d735-96db3ad20b864deb83dc3e2d64e2787e on tablet 96db3ad20b864deb83dc3e2d64e2787e with peer ab1d21935d6049888eb0ceccc3c9d735
I20240503 13:59:58.861263 15434 tablet_bootstrap.cc:492] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:58.861641 15434 tablet_bootstrap.cc:492] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap complete.
I20240503 13:59:58.861822 15434 ts_tablet_manager.cc:1367] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.034s user 0.024s sys 0.000s
I20240503 13:59:58.862286 15434 raft_consensus.cc:348] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 13:59:58.862433 15434 raft_consensus.cc:729] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: LEARNER
I20240503 13:59:58.862629 15434 consensus_queue.cc:260] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 13:59:58.862938 15434 ts_tablet_manager.cc:1398] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 13:59:58.864861 6942 tablet_copy_service.cc:342] P f58065db70b64ef8b1dc499aa6ae4460: Request end of tablet copy session 06ca8dbc004c42b99faa527a4e6f9545-7fde7d6bb07e4dbda5276b0dbbcff19b received from {username='slave'} at 127.0.99.133:60451
I20240503 13:59:58.864959 6942 tablet_copy_service.cc:433] P f58065db70b64ef8b1dc499aa6ae4460: ending tablet copy session 06ca8dbc004c42b99faa527a4e6f9545-7fde7d6bb07e4dbda5276b0dbbcff19b on tablet 7fde7d6bb07e4dbda5276b0dbbcff19b with peer 06ca8dbc004c42b99faa527a4e6f9545
I20240503 13:59:58.889182 15561 ts_tablet_manager.cc:902] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460: Initiating tablet copy from peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:58.889710 15561 tablet_copy_client.cc:308] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:41931
I20240503 13:59:58.890213 9362 tablet_copy_service.cc:140] P 0d77752181e944698e809e325e5937e6: Received BeginTabletCopySession request for tablet 1be8960eed8148f5958f6bb412c30a7e from peer f58065db70b64ef8b1dc499aa6ae4460 ({username='slave'} at 127.0.99.132:60419)
I20240503 13:59:58.890352 9362 tablet_copy_service.cc:161] P 0d77752181e944698e809e325e5937e6: Beginning new tablet copy session on tablet 1be8960eed8148f5958f6bb412c30a7e from peer f58065db70b64ef8b1dc499aa6ae4460 at {username='slave'} at 127.0.99.132:60419: session id = f58065db70b64ef8b1dc499aa6ae4460-1be8960eed8148f5958f6bb412c30a7e
I20240503 13:59:58.891322 9362 tablet_copy_source_session.cc:205] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:58.892331 15561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be8960eed8148f5958f6bb412c30a7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:58.894372 15561 tablet_copy_client.cc:783] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:58.894524 15561 tablet_copy_client.cc:647] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:58.895684 15561 tablet_copy_client.cc:523] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:58.897209 15561 tablet_bootstrap.cc:492] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:58.900982 15561 tablet_bootstrap.cc:492] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:58.901345 15561 tablet_bootstrap.cc:492] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap complete.
I20240503 13:59:58.901548 15561 ts_tablet_manager.cc:1367] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 13:59:58.902042 15561 raft_consensus.cc:348] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:58.902171 15561 raft_consensus.cc:729] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: LEARNER
I20240503 13:59:58.902277 15561 consensus_queue.cc:260] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:58.902578 15561 ts_tablet_manager.cc:1398] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:58.904327 9362 tablet_copy_service.cc:342] P 0d77752181e944698e809e325e5937e6: Request end of tablet copy session f58065db70b64ef8b1dc499aa6ae4460-1be8960eed8148f5958f6bb412c30a7e received from {username='slave'} at 127.0.99.132:60419
I20240503 13:59:58.904412 9362 tablet_copy_service.cc:433] P 0d77752181e944698e809e325e5937e6: ending tablet copy session f58065db70b64ef8b1dc499aa6ae4460-1be8960eed8148f5958f6bb412c30a7e on tablet 1be8960eed8148f5958f6bb412c30a7e with peer f58065db70b64ef8b1dc499aa6ae4460
I20240503 13:59:58.908331 9343 consensus_queue.cc:237] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:58.909612 6922 raft_consensus.cc:1261] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:58.909849 8946 raft_consensus.cc:1261] T a914c22ceff44fc685aad0fdcd04d844 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:58.910269 9278 consensus_peers.cc:485] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 -> Peer ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): Couldn't send request to peer ab1d21935d6049888eb0ceccc3c9d735. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a914c22ceff44fc685aad0fdcd04d844. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:58.910480 11378 consensus_queue.cc:1035] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:58.910663 14432 consensus_queue.cc:1035] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:58.912480 15169 raft_consensus.cc:2932] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:58.912879 8946 raft_consensus.cc:2932] T a914c22ceff44fc685aad0fdcd04d844 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:58.916620 6379 catalog_manager.cc:5484] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index -1 to 2, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:58.918267 6922 raft_consensus.cc:2932] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:58.920585 9080 raft_consensus.cc:1261] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 13:59:58.921118 15152 consensus_queue.cc:1035] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.929922 9210 raft_consensus.cc:1261] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 13:59:58.930456 15152 consensus_queue.cc:1035] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 13:59:58.937119 6527 raft_consensus.cc:1261] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 13:59:58.937546 12032 consensus_queue.cc:1035] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.941160 6922 consensus_queue.cc:237] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:58.943408 6658 raft_consensus.cc:1261] T db14294a5205460dbdd0f7588d92c1b5 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:59:58.943518 6857 consensus_peers.cc:485] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): Couldn't send request to peer 8d574fb4a89d4eea947c859470d44dd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db14294a5205460dbdd0f7588d92c1b5. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:58.943820 15111 consensus_queue.cc:1035] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.946321 6790 raft_consensus.cc:1261] T db14294a5205460dbdd0f7588d92c1b5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.948165 15111 consensus_queue.cc:1035] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.950976 13370 raft_consensus.cc:2932] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:58.951385 6788 raft_consensus.cc:2932] T db14294a5205460dbdd0f7588d92c1b5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:58.954986 6379 catalog_manager.cc:5484] T db14294a5205460dbdd0f7588d92c1b5 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index -1 to 4, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:58.958073 6658 raft_consensus.cc:2932] T db14294a5205460dbdd0f7588d92c1b5 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } } }
I20240503 13:59:58.966440 6788 consensus_queue.cc:237] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:58.967943 6658 raft_consensus.cc:1261] T ea438ff21d9744f697878406493f0fdc P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:58.968043 6726 consensus_peers.cc:485] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 -> Peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): Couldn't send request to peer f58065db70b64ef8b1dc499aa6ae4460. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea438ff21d9744f697878406493f0fdc. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:58.968287 6527 raft_consensus.cc:1261] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:58.968658 14880 consensus_queue.cc:1035] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:58.968854 14879 consensus_queue.cc:1035] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:58.970518 14879 raft_consensus.cc:2932] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:58.971670 6658 raft_consensus.cc:2932] T ea438ff21d9744f697878406493f0fdc P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:58.973461 6527 raft_consensus.cc:2932] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:58.978359 6385 catalog_manager.cc:5484] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index -1 to 2, NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } } }
I20240503 13:59:58.984438 6922 consensus_queue.cc:237] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 13:59:58.985945 9343 raft_consensus.cc:1261] T f403dc06195d449e8f454a9f4e795925 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.986076 6527 raft_consensus.cc:1261] T f403dc06195d449e8f454a9f4e795925 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:58.986387 15111 consensus_queue.cc:1035] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.986582 12748 consensus_queue.cc:1035] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:58.989317 6527 consensus_queue.cc:237] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:58.989343 15111 raft_consensus.cc:2932] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:58.990969 8946 raft_consensus.cc:1261] T a9e87083385d4e6bac9bda95670b68a8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:58.991447 9210 raft_consensus.cc:1261] T a9e87083385d4e6bac9bda95670b68a8 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:58.991433 15415 consensus_queue.cc:1035] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:58.992158 15416 consensus_queue.cc:1035] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 13:59:58.992158 6465 consensus_peers.cc:485] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9e87083385d4e6bac9bda95670b68a8. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:58.993783 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f403dc06195d449e8f454a9f4e795925 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:58.993865 15415 raft_consensus.cc:2932] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:58.994107 9343 raft_consensus.cc:2932] T f403dc06195d449e8f454a9f4e795925 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:58.995054 6379 catalog_manager.cc:5484] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 3 to 4, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:58.996846 9205 raft_consensus.cc:2932] T a9e87083385d4e6bac9bda95670b68a8 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:58.997431 6380 catalog_manager.cc:5484] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index -1 to 2, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:59.000785 6639 tablet_service.cc:1514] Processing DeleteTablet for tablet f403dc06195d449e8f454a9f4e795925 with delete_type TABLET_DATA_TOMBSTONED (TS 2145d15ab2d74c0287839266f1411a43 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37976
I20240503 13:59:59.001050 15371 tablet_replica.cc:330] T f403dc06195d449e8f454a9f4e795925 P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:59.001222 15371 raft_consensus.cc:2229] T f403dc06195d449e8f454a9f4e795925 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:59.001380 15371 raft_consensus.cc:2258] T f403dc06195d449e8f454a9f4e795925 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:59.002156 15371 ts_tablet_manager.cc:1875] T f403dc06195d449e8f454a9f4e795925 P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:59.004096 6527 raft_consensus.cc:2932] T f403dc06195d449e8f454a9f4e795925 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:59.004398 15371 ts_tablet_manager.cc:1888] T f403dc06195d449e8f454a9f4e795925 P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:59.004477 15371 log.cc:1196] T f403dc06195d449e8f454a9f4e795925 P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/f403dc06195d449e8f454a9f4e795925
I20240503 13:59:59.005084 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet f403dc06195d449e8f454a9f4e795925 (table rebalancer_extra_table_70 [id=0d758a66160e4c21a4e6c8f5b3cb6179]) successfully deleted
I20240503 13:59:59.010694 8946 raft_consensus.cc:2932] T a9e87083385d4e6bac9bda95670b68a8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:59.019470 9205 consensus_queue.cc:237] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:59.020879 8946 raft_consensus.cc:1261] T 568c2e7206f9496baa22432b72c021b1 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:59.020960 9080 raft_consensus.cc:1261] T 568c2e7206f9496baa22432b72c021b1 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:59.021332 14486 consensus_queue.cc:1035] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:59.021569 12063 consensus_queue.cc:1035] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 13:59:59.021775 9146 consensus_peers.cc:485] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 568c2e7206f9496baa22432b72c021b1. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:59.028543 12063 raft_consensus.cc:2932] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:59.030576 9080 raft_consensus.cc:2932] T 568c2e7206f9496baa22432b72c021b1 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:59.036373 6380 catalog_manager.cc:5484] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:59.037230 15444 ts_tablet_manager.cc:902] T 9a6709e4024d47448c20d33a2b36479e P 0d77752181e944698e809e325e5937e6: Initiating tablet copy from peer 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:59.037700 15444 tablet_copy_client.cc:308] T 9a6709e4024d47448c20d33a2b36479e P 0d77752181e944698e809e325e5937e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38021
I20240503 13:59:59.038417 6549 tablet_copy_service.cc:140] P 5f46e19644fd4f4b825c86ba6af768da: Received BeginTabletCopySession request for tablet 9a6709e4024d47448c20d33a2b36479e from peer 0d77752181e944698e809e325e5937e6 ({username='slave'} at 127.0.99.136:52973)
I20240503 13:59:59.038589 6549 tablet_copy_service.cc:161] P 5f46e19644fd4f4b825c86ba6af768da: Beginning new tablet copy session on tablet 9a6709e4024d47448c20d33a2b36479e from peer 0d77752181e944698e809e325e5937e6 at {username='slave'} at 127.0.99.136:52973: session id = 0d77752181e944698e809e325e5937e6-9a6709e4024d47448c20d33a2b36479e
I20240503 13:59:59.039779 6549 tablet_copy_source_session.cc:205] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:59.041543 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a6709e4024d47448c20d33a2b36479e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:59.044262 15444 tablet_copy_client.cc:783] T 9a6709e4024d47448c20d33a2b36479e P 0d77752181e944698e809e325e5937e6: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:59.044456 15444 tablet_copy_client.cc:647] T 9a6709e4024d47448c20d33a2b36479e P 0d77752181e944698e809e325e5937e6: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:59.047930 8946 raft_consensus.cc:2932] T 568c2e7206f9496baa22432b72c021b1 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 13:59:59.052656 15444 tablet_copy_client.cc:523] T 9a6709e4024d47448c20d33a2b36479e P 0d77752181e944698e809e325e5937e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:59.054021 15444 tablet_bootstrap.cc:492] T 9a6709e4024d47448c20d33a2b36479e P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:59.057144 15444 tablet_bootstrap.cc:492] T 9a6709e4024d47448c20d33a2b36479e P 0d77752181e944698e809e325e5937e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:59.057394 15444 tablet_bootstrap.cc:492] T 9a6709e4024d47448c20d33a2b36479e P 0d77752181e944698e809e325e5937e6: Bootstrap complete.
I20240503 13:59:59.057534 15444 ts_tablet_manager.cc:1367] T 9a6709e4024d47448c20d33a2b36479e P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 13:59:59.057931 15444 raft_consensus.cc:348] T 9a6709e4024d47448c20d33a2b36479e P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:59.058063 15444 raft_consensus.cc:729] T 9a6709e4024d47448c20d33a2b36479e P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: LEARNER
I20240503 13:59:59.058188 15444 consensus_queue.cc:260] T 9a6709e4024d47448c20d33a2b36479e P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:59.058444 15444 ts_tablet_manager.cc:1398] T 9a6709e4024d47448c20d33a2b36479e P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:59.060246 6549 tablet_copy_service.cc:342] P 5f46e19644fd4f4b825c86ba6af768da: Request end of tablet copy session 0d77752181e944698e809e325e5937e6-9a6709e4024d47448c20d33a2b36479e received from {username='slave'} at 127.0.99.136:52973
I20240503 13:59:59.060323 6549 tablet_copy_service.cc:433] P 5f46e19644fd4f4b825c86ba6af768da: ending tablet copy session 0d77752181e944698e809e325e5937e6-9a6709e4024d47448c20d33a2b36479e on tablet 9a6709e4024d47448c20d33a2b36479e with peer 0d77752181e944698e809e325e5937e6
I20240503 13:59:59.111645 15434 ts_tablet_manager.cc:902] T 9de4c2ac66344f719b3fe56ffbcc804d P 06ca8dbc004c42b99faa527a4e6f9545: Initiating tablet copy from peer 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 13:59:59.112257 15434 tablet_copy_client.cc:308] T 9de4c2ac66344f719b3fe56ffbcc804d P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46119
I20240503 13:59:59.112748 6680 tablet_copy_service.cc:140] P 2145d15ab2d74c0287839266f1411a43: Received BeginTabletCopySession request for tablet 9de4c2ac66344f719b3fe56ffbcc804d from peer 06ca8dbc004c42b99faa527a4e6f9545 ({username='slave'} at 127.0.99.133:36361)
I20240503 13:59:59.112880 6680 tablet_copy_service.cc:161] P 2145d15ab2d74c0287839266f1411a43: Beginning new tablet copy session on tablet 9de4c2ac66344f719b3fe56ffbcc804d from peer 06ca8dbc004c42b99faa527a4e6f9545 at {username='slave'} at 127.0.99.133:36361: session id = 06ca8dbc004c42b99faa527a4e6f9545-9de4c2ac66344f719b3fe56ffbcc804d
I20240503 13:59:59.113754 6680 tablet_copy_source_session.cc:205] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:59.114954 15434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9de4c2ac66344f719b3fe56ffbcc804d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:59.117296 15434 tablet_copy_client.cc:783] T 9de4c2ac66344f719b3fe56ffbcc804d P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:59.117465 15434 tablet_copy_client.cc:647] T 9de4c2ac66344f719b3fe56ffbcc804d P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:59.118811 15434 tablet_copy_client.cc:523] T 9de4c2ac66344f719b3fe56ffbcc804d P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:59.120177 15434 tablet_bootstrap.cc:492] T 9de4c2ac66344f719b3fe56ffbcc804d P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 13:59:59.134793 15508 ts_tablet_manager.cc:902] T 96d1a236cdb7406090021842d8dbc5a0 P 2145d15ab2d74c0287839266f1411a43: Initiating tablet copy from peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 13:59:59.135313 15508 tablet_copy_client.cc:308] T 96d1a236cdb7406090021842d8dbc5a0 P 2145d15ab2d74c0287839266f1411a43: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:34547
I20240503 13:59:59.135846 8968 tablet_copy_service.cc:140] P 06ca8dbc004c42b99faa527a4e6f9545: Received BeginTabletCopySession request for tablet 96d1a236cdb7406090021842d8dbc5a0 from peer 2145d15ab2d74c0287839266f1411a43 ({username='slave'} at 127.0.99.130:39847)
I20240503 13:59:59.135957 8968 tablet_copy_service.cc:161] P 06ca8dbc004c42b99faa527a4e6f9545: Beginning new tablet copy session on tablet 96d1a236cdb7406090021842d8dbc5a0 from peer 2145d15ab2d74c0287839266f1411a43 at {username='slave'} at 127.0.99.130:39847: session id = 2145d15ab2d74c0287839266f1411a43-96d1a236cdb7406090021842d8dbc5a0
I20240503 13:59:59.137118 8968 tablet_copy_source_session.cc:205] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:59.138183 15508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96d1a236cdb7406090021842d8dbc5a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:59.142817 15508 tablet_copy_client.cc:783] T 96d1a236cdb7406090021842d8dbc5a0 P 2145d15ab2d74c0287839266f1411a43: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:59.143158 15508 tablet_copy_client.cc:647] T 96d1a236cdb7406090021842d8dbc5a0 P 2145d15ab2d74c0287839266f1411a43: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:59.144466 15508 tablet_copy_client.cc:523] T 96d1a236cdb7406090021842d8dbc5a0 P 2145d15ab2d74c0287839266f1411a43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:59.145891 15508 tablet_bootstrap.cc:492] T 96d1a236cdb7406090021842d8dbc5a0 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 13:59:59.153563 15508 tablet_bootstrap.cc:492] T 96d1a236cdb7406090021842d8dbc5a0 P 2145d15ab2d74c0287839266f1411a43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:59.153947 15508 tablet_bootstrap.cc:492] T 96d1a236cdb7406090021842d8dbc5a0 P 2145d15ab2d74c0287839266f1411a43: Bootstrap complete.
I20240503 13:59:59.154119 15508 ts_tablet_manager.cc:1367] T 96d1a236cdb7406090021842d8dbc5a0 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240503 13:59:59.154592 15508 raft_consensus.cc:348] T 96d1a236cdb7406090021842d8dbc5a0 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:59.154729 15508 raft_consensus.cc:729] T 96d1a236cdb7406090021842d8dbc5a0 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: LEARNER
I20240503 13:59:59.154821 15508 consensus_queue.cc:260] T 96d1a236cdb7406090021842d8dbc5a0 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 13:59:59.155350 15508 ts_tablet_manager.cc:1398] T 96d1a236cdb7406090021842d8dbc5a0 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:59.156484 8968 tablet_copy_service.cc:342] P 06ca8dbc004c42b99faa527a4e6f9545: Request end of tablet copy session 2145d15ab2d74c0287839266f1411a43-96d1a236cdb7406090021842d8dbc5a0 received from {username='slave'} at 127.0.99.130:39847
I20240503 13:59:59.156582 8968 tablet_copy_service.cc:433] P 06ca8dbc004c42b99faa527a4e6f9545: ending tablet copy session 2145d15ab2d74c0287839266f1411a43-96d1a236cdb7406090021842d8dbc5a0 on tablet 96d1a236cdb7406090021842d8dbc5a0 with peer 2145d15ab2d74c0287839266f1411a43
I20240503 13:59:59.162837 15434 tablet_bootstrap.cc:492] T 9de4c2ac66344f719b3fe56ffbcc804d P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:59.163167 15434 tablet_bootstrap.cc:492] T 9de4c2ac66344f719b3fe56ffbcc804d P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap complete.
I20240503 13:59:59.163420 15434 ts_tablet_manager.cc:1367] T 9de4c2ac66344f719b3fe56ffbcc804d P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.043s user 0.027s sys 0.000s
I20240503 13:59:59.164021 15434 raft_consensus.cc:348] T 9de4c2ac66344f719b3fe56ffbcc804d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 13:59:59.164162 15434 raft_consensus.cc:729] T 9de4c2ac66344f719b3fe56ffbcc804d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: LEARNER
I20240503 13:59:59.164283 15434 consensus_queue.cc:260] T 9de4c2ac66344f719b3fe56ffbcc804d P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 13:59:59.167237 15434 ts_tablet_manager.cc:1398] T 9de4c2ac66344f719b3fe56ffbcc804d P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:59:59.168581 6680 tablet_copy_service.cc:342] P 2145d15ab2d74c0287839266f1411a43: Request end of tablet copy session 06ca8dbc004c42b99faa527a4e6f9545-9de4c2ac66344f719b3fe56ffbcc804d received from {username='slave'} at 127.0.99.133:36361
I20240503 13:59:59.168676 6680 tablet_copy_service.cc:433] P 2145d15ab2d74c0287839266f1411a43: ending tablet copy session 06ca8dbc004c42b99faa527a4e6f9545-9de4c2ac66344f719b3fe56ffbcc804d on tablet 9de4c2ac66344f719b3fe56ffbcc804d with peer 06ca8dbc004c42b99faa527a4e6f9545
I20240503 13:59:59.198944 6658 raft_consensus.cc:1203] T 2ed7d05584d14fdaafb625eaff3ecf08 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:59.221305 6788 consensus_queue.cc:237] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
W20240503 13:59:59.223018 6727 consensus_peers.cc:485] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d55d270456d3460586ff3cd41787b5d2. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:59.223004 8946 raft_consensus.cc:1203] T 7203f103a97141dcb70f42c0d27faa37 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:59.223145 6527 raft_consensus.cc:1261] T d55d270456d3460586ff3cd41787b5d2 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:59.223697 14880 consensus_queue.cc:1035] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:59.224195 6922 raft_consensus.cc:1261] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:59.225186 14880 consensus_queue.cc:1035] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:59.227531 12032 raft_consensus.cc:2932] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:59.227948 6922 raft_consensus.cc:2932] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:59.230840 6527 raft_consensus.cc:2932] T d55d270456d3460586ff3cd41787b5d2 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:59.234134 6380 catalog_manager.cc:5484] T d55d270456d3460586ff3cd41787b5d2 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 13:59:59.261060 9343 raft_consensus.cc:1203] T 342229593b184019ab915c0389b184b4 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:59.310142 15535 ts_tablet_manager.cc:902] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735: Initiating tablet copy from peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 13:59:59.310611 15535 tablet_copy_client.cc:308] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:41931
I20240503 13:59:59.311084 9362 tablet_copy_service.cc:140] P 0d77752181e944698e809e325e5937e6: Received BeginTabletCopySession request for tablet a914c22ceff44fc685aad0fdcd04d844 from peer ab1d21935d6049888eb0ceccc3c9d735 ({username='slave'} at 127.0.99.135:45341)
I20240503 13:59:59.311193 9362 tablet_copy_service.cc:161] P 0d77752181e944698e809e325e5937e6: Beginning new tablet copy session on tablet a914c22ceff44fc685aad0fdcd04d844 from peer ab1d21935d6049888eb0ceccc3c9d735 at {username='slave'} at 127.0.99.135:45341: session id = ab1d21935d6049888eb0ceccc3c9d735-a914c22ceff44fc685aad0fdcd04d844
I20240503 13:59:59.312204 9362 tablet_copy_source_session.cc:205] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:59.313210 15535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a914c22ceff44fc685aad0fdcd04d844. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:59.315811 15535 tablet_copy_client.cc:783] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:59.316071 15535 tablet_copy_client.cc:647] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:59.317807 15535 tablet_copy_client.cc:523] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:59.319628 15535 tablet_bootstrap.cc:492] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 13:59:59.323576 8946 raft_consensus.cc:1203] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:59.331001 15535 tablet_bootstrap.cc:492] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:59.331626 15535 tablet_bootstrap.cc:492] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap complete.
I20240503 13:59:59.331921 15535 ts_tablet_manager.cc:1367] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20240503 13:59:59.332429 15535 raft_consensus.cc:348] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:59.332613 15535 raft_consensus.cc:729] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: LEARNER
I20240503 13:59:59.332705 15535 consensus_queue.cc:260] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:59.332970 15535 ts_tablet_manager.cc:1398] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:59.334676 9362 tablet_copy_service.cc:342] P 0d77752181e944698e809e325e5937e6: Request end of tablet copy session ab1d21935d6049888eb0ceccc3c9d735-a914c22ceff44fc685aad0fdcd04d844 received from {username='slave'} at 127.0.99.135:45341
I20240503 13:59:59.334753 9362 tablet_copy_service.cc:433] P 0d77752181e944698e809e325e5937e6: ending tablet copy session ab1d21935d6049888eb0ceccc3c9d735-a914c22ceff44fc685aad0fdcd04d844 on tablet a914c22ceff44fc685aad0fdcd04d844 with peer ab1d21935d6049888eb0ceccc3c9d735
I20240503 13:59:59.345805 9205 raft_consensus.cc:1203] T 96db3ad20b864deb83dc3e2d64e2787e P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:59.376075 15561 ts_tablet_manager.cc:902] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460: Initiating tablet copy from peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:59.376542 15561 tablet_copy_client.cc:308] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:46365
I20240503 13:59:59.376997 6811 tablet_copy_service.cc:140] P 4a85e492c8154e9881a9cc2a105d7e64: Received BeginTabletCopySession request for tablet ea438ff21d9744f697878406493f0fdc from peer f58065db70b64ef8b1dc499aa6ae4460 ({username='slave'} at 127.0.99.132:42563)
I20240503 13:59:59.377094 6811 tablet_copy_service.cc:161] P 4a85e492c8154e9881a9cc2a105d7e64: Beginning new tablet copy session on tablet ea438ff21d9744f697878406493f0fdc from peer f58065db70b64ef8b1dc499aa6ae4460 at {username='slave'} at 127.0.99.132:42563: session id = f58065db70b64ef8b1dc499aa6ae4460-ea438ff21d9744f697878406493f0fdc
I20240503 13:59:59.377957 6811 tablet_copy_source_session.cc:205] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:59.378857 15561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea438ff21d9744f697878406493f0fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:59.381250 15561 tablet_copy_client.cc:783] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:59.381409 15561 tablet_copy_client.cc:647] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:59.382877 15561 tablet_copy_client.cc:523] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:59.383981 15561 tablet_bootstrap.cc:492] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap starting.
I20240503 13:59:59.387362 15822 ts_tablet_manager.cc:902] T db14294a5205460dbdd0f7588d92c1b5 P 8d574fb4a89d4eea947c859470d44dd4: Initiating tablet copy from peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 13:59:59.387435 15561 tablet_bootstrap.cc:492] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:59.387688 15561 tablet_bootstrap.cc:492] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460: Bootstrap complete.
I20240503 13:59:59.387837 15822 tablet_copy_client.cc:308] T db14294a5205460dbdd0f7588d92c1b5 P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43531
I20240503 13:59:59.387916 15561 ts_tablet_manager.cc:1367] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20240503 13:59:59.388288 6942 tablet_copy_service.cc:140] P f58065db70b64ef8b1dc499aa6ae4460: Received BeginTabletCopySession request for tablet db14294a5205460dbdd0f7588d92c1b5 from peer 8d574fb4a89d4eea947c859470d44dd4 ({username='slave'} at 127.0.99.134:57297)
I20240503 13:59:59.388382 6942 tablet_copy_service.cc:161] P f58065db70b64ef8b1dc499aa6ae4460: Beginning new tablet copy session on tablet db14294a5205460dbdd0f7588d92c1b5 from peer 8d574fb4a89d4eea947c859470d44dd4 at {username='slave'} at 127.0.99.134:57297: session id = 8d574fb4a89d4eea947c859470d44dd4-db14294a5205460dbdd0f7588d92c1b5
I20240503 13:59:59.388376 15561 raft_consensus.cc:348] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:59.388521 15561 raft_consensus.cc:729] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Initialized, Role: LEARNER
I20240503 13:59:59.388628 15561 consensus_queue.cc:260] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: true } }
I20240503 13:59:59.389252 15561 ts_tablet_manager.cc:1398] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:59.389505 6942 tablet_copy_source_session.cc:205] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:59.389818 6811 tablet_copy_service.cc:342] P 4a85e492c8154e9881a9cc2a105d7e64: Request end of tablet copy session f58065db70b64ef8b1dc499aa6ae4460-ea438ff21d9744f697878406493f0fdc received from {username='slave'} at 127.0.99.132:42563
I20240503 13:59:59.389887 6811 tablet_copy_service.cc:433] P 4a85e492c8154e9881a9cc2a105d7e64: ending tablet copy session f58065db70b64ef8b1dc499aa6ae4460-ea438ff21d9744f697878406493f0fdc on tablet ea438ff21d9744f697878406493f0fdc with peer f58065db70b64ef8b1dc499aa6ae4460
I20240503 13:59:59.391186 15822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db14294a5205460dbdd0f7588d92c1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:59.394418 15822 tablet_copy_client.cc:783] T db14294a5205460dbdd0f7588d92c1b5 P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:59.394580 15822 tablet_copy_client.cc:647] T db14294a5205460dbdd0f7588d92c1b5 P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:59.396082 15822 tablet_copy_client.cc:523] T db14294a5205460dbdd0f7588d92c1b5 P 8d574fb4a89d4eea947c859470d44dd4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:59.397948 15822 tablet_bootstrap.cc:492] T db14294a5205460dbdd0f7588d92c1b5 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap starting.
I20240503 13:59:59.400799 9343 raft_consensus.cc:1203] T 00434c9ea1654f7a9a9c31397a71998d P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:59.420315 15822 tablet_bootstrap.cc:492] T db14294a5205460dbdd0f7588d92c1b5 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:59.420629 15822 tablet_bootstrap.cc:492] T db14294a5205460dbdd0f7588d92c1b5 P 8d574fb4a89d4eea947c859470d44dd4: Bootstrap complete.
I20240503 13:59:59.420818 15822 ts_tablet_manager.cc:1367] T db14294a5205460dbdd0f7588d92c1b5 P 8d574fb4a89d4eea947c859470d44dd4: Time spent bootstrapping tablet: real 0.023s user 0.022s sys 0.000s
I20240503 13:59:59.421223 15822 raft_consensus.cc:348] T db14294a5205460dbdd0f7588d92c1b5 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:59.421329 15822 raft_consensus.cc:729] T db14294a5205460dbdd0f7588d92c1b5 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Initialized, Role: LEARNER
I20240503 13:59:59.421469 15822 consensus_queue.cc:260] T db14294a5205460dbdd0f7588d92c1b5 P 8d574fb4a89d4eea947c859470d44dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: true } }
I20240503 13:59:59.421764 15822 ts_tablet_manager.cc:1398] T db14294a5205460dbdd0f7588d92c1b5 P 8d574fb4a89d4eea947c859470d44dd4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:59.422256 6942 tablet_copy_service.cc:342] P f58065db70b64ef8b1dc499aa6ae4460: Request end of tablet copy session 8d574fb4a89d4eea947c859470d44dd4-db14294a5205460dbdd0f7588d92c1b5 received from {username='slave'} at 127.0.99.134:57297
I20240503 13:59:59.422346 6942 tablet_copy_service.cc:433] P f58065db70b64ef8b1dc499aa6ae4460: ending tablet copy session 8d574fb4a89d4eea947c859470d44dd4-db14294a5205460dbdd0f7588d92c1b5 on tablet db14294a5205460dbdd0f7588d92c1b5 with peer 8d574fb4a89d4eea947c859470d44dd4
I20240503 13:59:59.432960 15444 ts_tablet_manager.cc:902] T a9e87083385d4e6bac9bda95670b68a8 P 0d77752181e944698e809e325e5937e6: Initiating tablet copy from peer 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 13:59:59.433450 15444 tablet_copy_client.cc:308] T a9e87083385d4e6bac9bda95670b68a8 P 0d77752181e944698e809e325e5937e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38021
I20240503 13:59:59.433873 6549 tablet_copy_service.cc:140] P 5f46e19644fd4f4b825c86ba6af768da: Received BeginTabletCopySession request for tablet a9e87083385d4e6bac9bda95670b68a8 from peer 0d77752181e944698e809e325e5937e6 ({username='slave'} at 127.0.99.136:52973)
I20240503 13:59:59.433982 6549 tablet_copy_service.cc:161] P 5f46e19644fd4f4b825c86ba6af768da: Beginning new tablet copy session on tablet a9e87083385d4e6bac9bda95670b68a8 from peer 0d77752181e944698e809e325e5937e6 at {username='slave'} at 127.0.99.136:52973: session id = 0d77752181e944698e809e325e5937e6-a9e87083385d4e6bac9bda95670b68a8
I20240503 13:59:59.434810 6549 tablet_copy_source_session.cc:205] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:59.435674 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e87083385d4e6bac9bda95670b68a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:59.437618 15444 tablet_copy_client.cc:783] T a9e87083385d4e6bac9bda95670b68a8 P 0d77752181e944698e809e325e5937e6: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:59.437780 15444 tablet_copy_client.cc:647] T a9e87083385d4e6bac9bda95670b68a8 P 0d77752181e944698e809e325e5937e6: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:59.438997 15444 tablet_copy_client.cc:523] T a9e87083385d4e6bac9bda95670b68a8 P 0d77752181e944698e809e325e5937e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:59.440482 15444 tablet_bootstrap.cc:492] T a9e87083385d4e6bac9bda95670b68a8 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:59.444123 15444 tablet_bootstrap.cc:492] T a9e87083385d4e6bac9bda95670b68a8 P 0d77752181e944698e809e325e5937e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:59.444320 15444 tablet_bootstrap.cc:492] T a9e87083385d4e6bac9bda95670b68a8 P 0d77752181e944698e809e325e5937e6: Bootstrap complete.
I20240503 13:59:59.444482 15444 ts_tablet_manager.cc:1367] T a9e87083385d4e6bac9bda95670b68a8 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 13:59:59.444876 15444 raft_consensus.cc:348] T a9e87083385d4e6bac9bda95670b68a8 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:59.444976 15444 raft_consensus.cc:729] T a9e87083385d4e6bac9bda95670b68a8 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: LEARNER
I20240503 13:59:59.445044 15444 consensus_queue.cc:260] T a9e87083385d4e6bac9bda95670b68a8 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:59.445284 15444 ts_tablet_manager.cc:1398] T a9e87083385d4e6bac9bda95670b68a8 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:59.445796 6549 tablet_copy_service.cc:342] P 5f46e19644fd4f4b825c86ba6af768da: Request end of tablet copy session 0d77752181e944698e809e325e5937e6-a9e87083385d4e6bac9bda95670b68a8 received from {username='slave'} at 127.0.99.136:52973
I20240503 13:59:59.445873 6549 tablet_copy_service.cc:433] P 5f46e19644fd4f4b825c86ba6af768da: ending tablet copy session 0d77752181e944698e809e325e5937e6-a9e87083385d4e6bac9bda95670b68a8 on tablet a9e87083385d4e6bac9bda95670b68a8 with peer 0d77752181e944698e809e325e5937e6
I20240503 13:59:59.482299 15833 ts_tablet_manager.cc:902] T 568c2e7206f9496baa22432b72c021b1 P 4a85e492c8154e9881a9cc2a105d7e64: Initiating tablet copy from peer ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 13:59:59.482867 15833 tablet_copy_client.cc:308] T 568c2e7206f9496baa22432b72c021b1 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:34091
I20240503 13:59:59.483834 9343 raft_consensus.cc:1203] T 9a6709e4024d47448c20d33a2b36479e P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:59.485106 9231 tablet_copy_service.cc:140] P ab1d21935d6049888eb0ceccc3c9d735: Received BeginTabletCopySession request for tablet 568c2e7206f9496baa22432b72c021b1 from peer 4a85e492c8154e9881a9cc2a105d7e64 ({username='slave'} at 127.0.99.131:60171)
I20240503 13:59:59.485219 9231 tablet_copy_service.cc:161] P ab1d21935d6049888eb0ceccc3c9d735: Beginning new tablet copy session on tablet 568c2e7206f9496baa22432b72c021b1 from peer 4a85e492c8154e9881a9cc2a105d7e64 at {username='slave'} at 127.0.99.131:60171: session id = 4a85e492c8154e9881a9cc2a105d7e64-568c2e7206f9496baa22432b72c021b1
I20240503 13:59:59.486219 9231 tablet_copy_source_session.cc:205] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:59.487881 15833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568c2e7206f9496baa22432b72c021b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:59.490348 15833 tablet_copy_client.cc:783] T 568c2e7206f9496baa22432b72c021b1 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:59.490525 15833 tablet_copy_client.cc:647] T 568c2e7206f9496baa22432b72c021b1 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:59.492120 15833 tablet_copy_client.cc:523] T 568c2e7206f9496baa22432b72c021b1 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:59.493455 15833 tablet_bootstrap.cc:492] T 568c2e7206f9496baa22432b72c021b1 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 13:59:59.496855 15833 tablet_bootstrap.cc:492] T 568c2e7206f9496baa22432b72c021b1 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:59.497071 15833 tablet_bootstrap.cc:492] T 568c2e7206f9496baa22432b72c021b1 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap complete.
I20240503 13:59:59.497201 15833 ts_tablet_manager.cc:1367] T 568c2e7206f9496baa22432b72c021b1 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 13:59:59.497552 15833 raft_consensus.cc:348] T 568c2e7206f9496baa22432b72c021b1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:59.497656 15833 raft_consensus.cc:729] T 568c2e7206f9496baa22432b72c021b1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: LEARNER
I20240503 13:59:59.497771 15833 consensus_queue.cc:260] T 568c2e7206f9496baa22432b72c021b1 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 13:59:59.498062 15833 ts_tablet_manager.cc:1398] T 568c2e7206f9496baa22432b72c021b1 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:59.498592 9231 tablet_copy_service.cc:342] P ab1d21935d6049888eb0ceccc3c9d735: Request end of tablet copy session 4a85e492c8154e9881a9cc2a105d7e64-568c2e7206f9496baa22432b72c021b1 received from {username='slave'} at 127.0.99.131:60171
I20240503 13:59:59.498682 9231 tablet_copy_service.cc:433] P ab1d21935d6049888eb0ceccc3c9d735: ending tablet copy session 4a85e492c8154e9881a9cc2a105d7e64-568c2e7206f9496baa22432b72c021b1 on tablet 568c2e7206f9496baa22432b72c021b1 with peer 4a85e492c8154e9881a9cc2a105d7e64
I20240503 13:59:59.500535 6922 raft_consensus.cc:1203] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:59.546600 6788 consensus_queue.cc:237] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 13:59:59.548235 9205 raft_consensus.cc:1261] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:59:59.548308 6527 raft_consensus.cc:1261] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:59:59.548713 14879 consensus_queue.cc:1035] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:59.548955 15152 consensus_queue.cc:1035] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:59.550624 14879 raft_consensus.cc:2932] T 387d78763d574e57b914e2dbac83a932 P 4a85e492c8154e9881a9cc2a105d7e64 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:59.551108 9205 raft_consensus.cc:2932] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:59.551152 6527 raft_consensus.cc:2932] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:59.552505 6372 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 387d78763d574e57b914e2dbac83a932 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:59.556000 6379 catalog_manager.cc:5484] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index 3 to 5, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New cstate: current_term: 2 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } } }
I20240503 13:59:59.558912 9060 tablet_service.cc:1514] Processing DeleteTablet for tablet 387d78763d574e57b914e2dbac83a932 with delete_type TABLET_DATA_TOMBSTONED (TS 8d574fb4a89d4eea947c859470d44dd4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:59.559497 15839 tablet_replica.cc:330] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:59.559657 15839 raft_consensus.cc:2229] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:59.559775 15839 raft_consensus.cc:2258] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:59.560395 15839 ts_tablet_manager.cc:1875] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:59.562377 15839 ts_tablet_manager.cc:1888] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 13:59:59.562445 15839 log.cc:1196] T 387d78763d574e57b914e2dbac83a932 P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/387d78763d574e57b914e2dbac83a932
I20240503 13:59:59.562914 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet 387d78763d574e57b914e2dbac83a932 (table rebalancer_extra_table_62 [id=550b2587d1ad46df98279b7c7b5fa594]) successfully deleted
I20240503 13:59:59.624434 14177 raft_consensus.cc:1050] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43: attempting to promote NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 to VOTER
I20240503 13:59:59.624861 14177 consensus_queue.cc:237] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
I20240503 13:59:59.626513 6527 raft_consensus.cc:1261] T 7203f103a97141dcb70f42c0d27faa37 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:59.626806 8947 raft_consensus.cc:1261] T 7203f103a97141dcb70f42c0d27faa37 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:59.627130 6788 raft_consensus.cc:1261] T 7203f103a97141dcb70f42c0d27faa37 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:59.627548 15432 consensus_queue.cc:1035] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:59.627740 15101 consensus_queue.cc:1035] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:59.628057 15876 consensus_queue.cc:1035] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:59.630434 15101 raft_consensus.cc:2932] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:59.631006 8947 raft_consensus.cc:2932] T 7203f103a97141dcb70f42c0d27faa37 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:59.632794 6527 raft_consensus.cc:2932] T 7203f103a97141dcb70f42c0d27faa37 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:59.634662 6380 catalog_manager.cc:5484] T 7203f103a97141dcb70f42c0d27faa37 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:59.637733 6788 raft_consensus.cc:2932] T 7203f103a97141dcb70f42c0d27faa37 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:59.638527 6659 consensus_queue.cc:237] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
I20240503 13:59:59.640342 6526 raft_consensus.cc:1261] T 7203f103a97141dcb70f42c0d27faa37 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:59.640753 15876 consensus_queue.cc:1035] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:59.641672 8947 raft_consensus.cc:1261] T 7203f103a97141dcb70f42c0d27faa37 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:59.642699 15876 consensus_queue.cc:1035] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:59.643024 14177 raft_consensus.cc:2932] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:59.644891 6374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7203f103a97141dcb70f42c0d27faa37 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:59.647028 8946 raft_consensus.cc:2932] T 7203f103a97141dcb70f42c0d27faa37 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:59.647723 6527 raft_consensus.cc:2932] T 7203f103a97141dcb70f42c0d27faa37 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:59.649093 6387 catalog_manager.cc:5484] T 7203f103a97141dcb70f42c0d27faa37 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index 5 to 6, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:59.652594 6769 tablet_service.cc:1514] Processing DeleteTablet for tablet 7203f103a97141dcb70f42c0d27faa37 with delete_type TABLET_DATA_TOMBSTONED (TS 4a85e492c8154e9881a9cc2a105d7e64 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:59.653118 15890 tablet_replica.cc:330] T 7203f103a97141dcb70f42c0d27faa37 P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:59.654933 6659 raft_consensus.cc:1203] T 96d1a236cdb7406090021842d8dbc5a0 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:59.656960 15890 raft_consensus.cc:2229] T 7203f103a97141dcb70f42c0d27faa37 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:59.657204 15890 raft_consensus.cc:2258] T 7203f103a97141dcb70f42c0d27faa37 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:59.658335 15890 ts_tablet_manager.cc:1875] T 7203f103a97141dcb70f42c0d27faa37 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:59.662756 15890 ts_tablet_manager.cc:1888] T 7203f103a97141dcb70f42c0d27faa37 P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:59.662849 15890 log.cc:1196] T 7203f103a97141dcb70f42c0d27faa37 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/7203f103a97141dcb70f42c0d27faa37
I20240503 13:59:59.664189 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet 7203f103a97141dcb70f42c0d27faa37 (table rebalance_test_table_5 [id=cb716c6127694172a7a76a19a23dc99e]) successfully deleted
I20240503 13:59:59.712525 12748 raft_consensus.cc:1050] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460: attempting to promote NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 to VOTER
I20240503 13:59:59.712965 12748 consensus_queue.cc:237] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
I20240503 13:59:59.714279 8946 raft_consensus.cc:1261] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:59.714423 6659 raft_consensus.cc:1261] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:59.714789 6788 raft_consensus.cc:1261] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:59.714911 15927 consensus_queue.cc:1035] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:59.715095 15111 consensus_queue.cc:1035] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:59.715226 12748 consensus_queue.cc:1035] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:59.718385 13370 raft_consensus.cc:2932] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:59.719249 6788 raft_consensus.cc:2932] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:59.720458 6659 raft_consensus.cc:2932] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:59.723399 6380 catalog_manager.cc:5484] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:59.725970 8946 raft_consensus.cc:2932] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:59.727048 6921 consensus_queue.cc:237] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
I20240503 13:59:59.728684 8946 raft_consensus.cc:1203] T 9de4c2ac66344f719b3fe56ffbcc804d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:59.729713 8946 raft_consensus.cc:1261] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:59.730415 15927 consensus_queue.cc:1035] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:59.732667 6790 raft_consensus.cc:1261] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:59.733348 15111 raft_consensus.cc:2932] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:59.733552 15927 consensus_queue.cc:1035] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:59.733930 8946 raft_consensus.cc:2932] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:59.734534 6790 raft_consensus.cc:2932] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:59.735831 12367 raft_consensus.cc:1050] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545: attempting to promote NON_VOTER 2145d15ab2d74c0287839266f1411a43 to VOTER
I20240503 13:59:59.736173 12367 consensus_queue.cc:237] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 13:59:59.737370 6788 raft_consensus.cc:1261] T 2ed7d05584d14fdaafb625eaff3ecf08 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:59.737705 9080 raft_consensus.cc:1261] T 2ed7d05584d14fdaafb625eaff3ecf08 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:59.737931 14390 consensus_queue.cc:1035] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:59.739300 6659 raft_consensus.cc:1261] T 2ed7d05584d14fdaafb625eaff3ecf08 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:59.739682 14384 consensus_queue.cc:1035] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:59.740010 14384 consensus_queue.cc:1035] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:59.742766 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7fde7d6bb07e4dbda5276b0dbbcff19b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:59.745684 6379 catalog_manager.cc:5484] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index 5 to 6, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 13:59:59.747273 12367 raft_consensus.cc:2932] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:59.749259 9080 raft_consensus.cc:2932] T 2ed7d05584d14fdaafb625eaff3ecf08 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:59.750083 6659 raft_consensus.cc:2932] T 2ed7d05584d14fdaafb625eaff3ecf08 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:59.753815 6385 catalog_manager.cc:5484] T 2ed7d05584d14fdaafb625eaff3ecf08 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:59.754514 8946 consensus_queue.cc:237] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 13:59:59.755656 6921 raft_consensus.cc:1203] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:59.755764 6639 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fde7d6bb07e4dbda5276b0dbbcff19b with delete_type TABLET_DATA_TOMBSTONED (TS 2145d15ab2d74c0287839266f1411a43 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37976
W20240503 13:59:59.757494 8883 consensus_peers.cc:485] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): Couldn't send request to peer ab1d21935d6049888eb0ceccc3c9d735. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d51634c08e2749d4a3bbff4b2a786e48. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:59.757956 6921 raft_consensus.cc:1261] T d51634c08e2749d4a3bbff4b2a786e48 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:59.758067 9080 raft_consensus.cc:1261] T d51634c08e2749d4a3bbff4b2a786e48 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:59.759235 14390 consensus_queue.cc:1035] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:59.759577 14390 consensus_queue.cc:1035] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:59.760689 15946 tablet_replica.cc:330] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 13:59:59.760829 15946 raft_consensus.cc:2229] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:59.760983 15946 raft_consensus.cc:2258] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:59.761688 15946 ts_tablet_manager.cc:1875] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:59.764420 6788 raft_consensus.cc:2932] T 2ed7d05584d14fdaafb625eaff3ecf08 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:59.764940 15946 ts_tablet_manager.cc:1888] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:59.767057 15946 log.cc:1196] T 7fde7d6bb07e4dbda5276b0dbbcff19b P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/7fde7d6bb07e4dbda5276b0dbbcff19b
I20240503 13:59:59.768021 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet 7fde7d6bb07e4dbda5276b0dbbcff19b (table rebalance_test_table_17 [id=f8de54141608449ab9dffc176985cb69]) successfully deleted
I20240503 13:59:59.768085 12367 raft_consensus.cc:2932] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:59.769770 9080 raft_consensus.cc:2932] T d51634c08e2749d4a3bbff4b2a786e48 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:59.772338 8946 consensus_queue.cc:237] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 13:59:59.774699 6387 catalog_manager.cc:5484] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index -1 to 2, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:59.775820 6788 raft_consensus.cc:1261] T 2ed7d05584d14fdaafb625eaff3ecf08 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:59.775964 6659 raft_consensus.cc:1261] T 2ed7d05584d14fdaafb625eaff3ecf08 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:59.776208 14390 consensus_queue.cc:1035] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:59.776611 14390 consensus_queue.cc:1035] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:59.777469 6921 raft_consensus.cc:2932] T d51634c08e2749d4a3bbff4b2a786e48 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } } }
I20240503 13:59:59.779111 14390 raft_consensus.cc:2932] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:59.781405 6658 raft_consensus.cc:2932] T 2ed7d05584d14fdaafb625eaff3ecf08 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:59.782327 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2ed7d05584d14fdaafb625eaff3ecf08 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:59.782527 6788 raft_consensus.cc:2932] T 2ed7d05584d14fdaafb625eaff3ecf08 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:59.785391 6379 catalog_manager.cc:5484] T 2ed7d05584d14fdaafb625eaff3ecf08 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 13:59:59.786214 14177 raft_consensus.cc:1050] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43: attempting to promote NON_VOTER 0d77752181e944698e809e325e5937e6 to VOTER
I20240503 13:59:59.786592 14177 consensus_queue.cc:237] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 13:59:59.788287 6921 raft_consensus.cc:1261] T 00434c9ea1654f7a9a9c31397a71998d P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:59.789005 6788 raft_consensus.cc:1261] T 00434c9ea1654f7a9a9c31397a71998d P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:59.789211 15432 consensus_queue.cc:1035] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:59.790637 9343 raft_consensus.cc:1261] T 00434c9ea1654f7a9a9c31397a71998d P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:59.790975 9060 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ed7d05584d14fdaafb625eaff3ecf08 with delete_type TABLET_DATA_TOMBSTONED (TS 8d574fb4a89d4eea947c859470d44dd4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:59.791098 15101 consensus_queue.cc:1035] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:59.791435 15101 consensus_queue.cc:1035] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:59.791590 15839 tablet_replica.cc:330] T 2ed7d05584d14fdaafb625eaff3ecf08 P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:59.791759 15839 raft_consensus.cc:2229] T 2ed7d05584d14fdaafb625eaff3ecf08 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:59.791911 15839 raft_consensus.cc:2258] T 2ed7d05584d14fdaafb625eaff3ecf08 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:59.793293 15839 ts_tablet_manager.cc:1875] T 2ed7d05584d14fdaafb625eaff3ecf08 P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:59.793246 6659 consensus_queue.cc:237] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 13:59:59.793947 14177 raft_consensus.cc:2932] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:59.795264 9343 raft_consensus.cc:2932] T 00434c9ea1654f7a9a9c31397a71998d P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:59.795504 15839 ts_tablet_manager.cc:1888] T 2ed7d05584d14fdaafb625eaff3ecf08 P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:59.795563 15839 log.cc:1196] T 2ed7d05584d14fdaafb625eaff3ecf08 P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/2ed7d05584d14fdaafb625eaff3ecf08
I20240503 13:59:59.797178 6921 raft_consensus.cc:1261] T b6cb5119a6db403ba3cae9fa9b121211 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:59.797577 6528 raft_consensus.cc:1261] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:59.798166 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet 2ed7d05584d14fdaafb625eaff3ecf08 (table rebalancer_extra_table_54 [id=0179e5ae38524fc9b8f37ce24d941401]) successfully deleted
W20240503 13:59:59.798189 6595 consensus_peers.cc:485] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 -> Peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): Couldn't send request to peer 06ca8dbc004c42b99faa527a4e6f9545. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6cb5119a6db403ba3cae9fa9b121211. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:59.798516 6921 raft_consensus.cc:2932] T 00434c9ea1654f7a9a9c31397a71998d P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:59.799645 15432 consensus_queue.cc:1035] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:59.799984 6387 catalog_manager.cc:5484] T 00434c9ea1654f7a9a9c31397a71998d P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:59.800593 15432 consensus_queue.cc:1035] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:59.804551 15959 raft_consensus.cc:2932] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:59.807432 6528 raft_consensus.cc:2932] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:59.808051 6790 raft_consensus.cc:2932] T 00434c9ea1654f7a9a9c31397a71998d P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:59.811079 6659 consensus_queue.cc:237] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 13:59:59.811901 6920 raft_consensus.cc:2932] T b6cb5119a6db403ba3cae9fa9b121211 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:59.812134 6385 catalog_manager.cc:5484] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 13:59:59.813000 9343 raft_consensus.cc:1261] T 00434c9ea1654f7a9a9c31397a71998d P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:59.813261 15444 ts_tablet_manager.cc:902] T d55d270456d3460586ff3cd41787b5d2 P 0d77752181e944698e809e325e5937e6: Initiating tablet copy from peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 13:59:59.813758 15444 tablet_copy_client.cc:308] T d55d270456d3460586ff3cd41787b5d2 P 0d77752181e944698e809e325e5937e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:46365
I20240503 13:59:59.814188 6790 raft_consensus.cc:1261] T 00434c9ea1654f7a9a9c31397a71998d P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:59.814507 6811 tablet_copy_service.cc:140] P 4a85e492c8154e9881a9cc2a105d7e64: Received BeginTabletCopySession request for tablet d55d270456d3460586ff3cd41787b5d2 from peer 0d77752181e944698e809e325e5937e6 ({username='slave'} at 127.0.99.136:34769)
I20240503 13:59:59.814600 6811 tablet_copy_service.cc:161] P 4a85e492c8154e9881a9cc2a105d7e64: Beginning new tablet copy session on tablet d55d270456d3460586ff3cd41787b5d2 from peer 0d77752181e944698e809e325e5937e6 at {username='slave'} at 127.0.99.136:34769: session id = 0d77752181e944698e809e325e5937e6-d55d270456d3460586ff3cd41787b5d2
I20240503 13:59:59.814661 15959 consensus_queue.cc:1035] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:59.814968 15876 consensus_queue.cc:1035] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:59.815742 6811 tablet_copy_source_session.cc:205] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:59.816668 15243 raft_consensus.cc:1050] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da: attempting to promote NON_VOTER 0d77752181e944698e809e325e5937e6 to VOTER
I20240503 13:59:59.817066 15243 consensus_queue.cc:237] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 13:59:59.817211 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55d270456d3460586ff3cd41787b5d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:59.818783 6920 raft_consensus.cc:1261] T 342229593b184019ab915c0389b184b4 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:59.819093 9342 raft_consensus.cc:1261] T 342229593b184019ab915c0389b184b4 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:59.819252 15099 consensus_queue.cc:1035] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:59.819996 15099 consensus_queue.cc:1035] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:59.822324 6788 raft_consensus.cc:1261] T 342229593b184019ab915c0389b184b4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:59.822749 15099 consensus_queue.cc:1035] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:59.825222 15432 raft_consensus.cc:2932] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:59.827193 9343 raft_consensus.cc:2932] T 00434c9ea1654f7a9a9c31397a71998d P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:59.827901 6374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 00434c9ea1654f7a9a9c31397a71998d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:59.828183 6790 raft_consensus.cc:2932] T 00434c9ea1654f7a9a9c31397a71998d P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:59.828460 15444 tablet_copy_client.cc:783] T d55d270456d3460586ff3cd41787b5d2 P 0d77752181e944698e809e325e5937e6: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:59.828608 15444 tablet_copy_client.cc:647] T d55d270456d3460586ff3cd41787b5d2 P 0d77752181e944698e809e325e5937e6: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:59.829936 6385 catalog_manager.cc:5484] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index 5 to 6, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:59.832662 15099 raft_consensus.cc:2932] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:59.833354 9342 raft_consensus.cc:2932] T 342229593b184019ab915c0389b184b4 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:59.835162 6900 tablet_service.cc:1514] Processing DeleteTablet for tablet 00434c9ea1654f7a9a9c31397a71998d with delete_type TABLET_DATA_TOMBSTONED (TS f58065db70b64ef8b1dc499aa6ae4460 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:59.835412 6920 raft_consensus.cc:2932] T 342229593b184019ab915c0389b184b4 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:59.838428 6788 raft_consensus.cc:2932] T 342229593b184019ab915c0389b184b4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:59.838737 6380 catalog_manager.cc:5484] T 342229593b184019ab915c0389b184b4 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 4 to 5, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:59.840773 15976 tablet_replica.cc:330] T 00434c9ea1654f7a9a9c31397a71998d P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:59.840992 15976 raft_consensus.cc:2229] T 00434c9ea1654f7a9a9c31397a71998d P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:59.841153 15976 raft_consensus.cc:2258] T 00434c9ea1654f7a9a9c31397a71998d P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:59.841585 15444 tablet_copy_client.cc:523] T d55d270456d3460586ff3cd41787b5d2 P 0d77752181e944698e809e325e5937e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:59.842108 15976 ts_tablet_manager.cc:1875] T 00434c9ea1654f7a9a9c31397a71998d P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:59.843065 15444 tablet_bootstrap.cc:492] T d55d270456d3460586ff3cd41787b5d2 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 13:59:59.846747 6527 consensus_queue.cc:237] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 13:59:59.848507 6788 raft_consensus.cc:1261] T 342229593b184019ab915c0389b184b4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:59.848773 9342 raft_consensus.cc:1261] T 342229593b184019ab915c0389b184b4 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:59.849224 15243 consensus_queue.cc:1035] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:59.849423 15972 consensus_queue.cc:1035] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:59.851157 15976 ts_tablet_manager.cc:1888] T 00434c9ea1654f7a9a9c31397a71998d P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:59.851300 15976 log.cc:1196] T 00434c9ea1654f7a9a9c31397a71998d P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/00434c9ea1654f7a9a9c31397a71998d
I20240503 13:59:59.851941 15416 raft_consensus.cc:2932] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:59.852191 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet 00434c9ea1654f7a9a9c31397a71998d (table rebalance_test_table_17 [id=f8de54141608449ab9dffc176985cb69]) successfully deleted
I20240503 13:59:59.852697 9342 raft_consensus.cc:2932] T 342229593b184019ab915c0389b184b4 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:59.854807 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 342229593b184019ab915c0389b184b4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:59.855847 6788 raft_consensus.cc:2932] T 342229593b184019ab915c0389b184b4 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 13:59:59.856560 6387 catalog_manager.cc:5484] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index 5 to 6, VOTER f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:59.859045 6900 tablet_service.cc:1514] Processing DeleteTablet for tablet 342229593b184019ab915c0389b184b4 with delete_type TABLET_DATA_TOMBSTONED (TS f58065db70b64ef8b1dc499aa6ae4460 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50722
I20240503 13:59:59.859228 15976 tablet_replica.cc:330] T 342229593b184019ab915c0389b184b4 P f58065db70b64ef8b1dc499aa6ae4460: stopping tablet replica
I20240503 13:59:59.859447 15976 raft_consensus.cc:2229] T 342229593b184019ab915c0389b184b4 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:59.859601 15976 raft_consensus.cc:2258] T 342229593b184019ab915c0389b184b4 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:59.860441 15976 ts_tablet_manager.cc:1875] T 342229593b184019ab915c0389b184b4 P f58065db70b64ef8b1dc499aa6ae4460: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:59.862393 15976 ts_tablet_manager.cc:1888] T 342229593b184019ab915c0389b184b4 P f58065db70b64ef8b1dc499aa6ae4460: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:59.862457 15976 log.cc:1196] T 342229593b184019ab915c0389b184b4 P f58065db70b64ef8b1dc499aa6ae4460: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/342229593b184019ab915c0389b184b4
I20240503 13:59:59.862970 6373 catalog_manager.cc:4830] TS f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531): tablet 342229593b184019ab915c0389b184b4 (table rebalance_test_table_11 [id=3e15c9008ce84692a697ae9651a18574]) successfully deleted
I20240503 13:59:59.875972 15444 tablet_bootstrap.cc:492] T d55d270456d3460586ff3cd41787b5d2 P 0d77752181e944698e809e325e5937e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:59.876271 15444 tablet_bootstrap.cc:492] T d55d270456d3460586ff3cd41787b5d2 P 0d77752181e944698e809e325e5937e6: Bootstrap complete.
I20240503 13:59:59.876461 15444 ts_tablet_manager.cc:1367] T d55d270456d3460586ff3cd41787b5d2 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.033s user 0.024s sys 0.000s
I20240503 13:59:59.877115 15444 raft_consensus.cc:348] T d55d270456d3460586ff3cd41787b5d2 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:59.877219 15444 raft_consensus.cc:729] T d55d270456d3460586ff3cd41787b5d2 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: LEARNER
I20240503 13:59:59.877321 15444 consensus_queue.cc:260] T d55d270456d3460586ff3cd41787b5d2 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 13:59:59.877594 15444 ts_tablet_manager.cc:1398] T d55d270456d3460586ff3cd41787b5d2 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:59.878018 6811 tablet_copy_service.cc:342] P 4a85e492c8154e9881a9cc2a105d7e64: Request end of tablet copy session 0d77752181e944698e809e325e5937e6-d55d270456d3460586ff3cd41787b5d2 received from {username='slave'} at 127.0.99.136:34769
I20240503 13:59:59.878103 6811 tablet_copy_service.cc:433] P 4a85e492c8154e9881a9cc2a105d7e64: ending tablet copy session 0d77752181e944698e809e325e5937e6-d55d270456d3460586ff3cd41787b5d2 on tablet d55d270456d3460586ff3cd41787b5d2 with peer 0d77752181e944698e809e325e5937e6
I20240503 13:59:59.901901 15796 raft_consensus.cc:1050] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4: attempting to promote NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 to VOTER
I20240503 13:59:59.902336 15796 consensus_queue.cc:237] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 13:59:59.903821 6659 raft_consensus.cc:1261] T 96db3ad20b864deb83dc3e2d64e2787e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:59.903967 6788 raft_consensus.cc:1261] T 96db3ad20b864deb83dc3e2d64e2787e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:59.904529 13280 consensus_queue.cc:1035] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:59.904697 16009 consensus_queue.cc:1035] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:59.904927 9210 raft_consensus.cc:1203] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:59.907806 9205 raft_consensus.cc:1261] T 96db3ad20b864deb83dc3e2d64e2787e P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:59.908710 16009 consensus_queue.cc:1035] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:59.910132 13280 raft_consensus.cc:2932] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:59.911191 9205 raft_consensus.cc:2932] T 96db3ad20b864deb83dc3e2d64e2787e P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:59.913208 6659 raft_consensus.cc:2932] T 96db3ad20b864deb83dc3e2d64e2787e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:59.915251 6387 catalog_manager.cc:5484] T 96db3ad20b864deb83dc3e2d64e2787e P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index 2 to 3, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:59.918144 6790 raft_consensus.cc:2932] T 96db3ad20b864deb83dc3e2d64e2787e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:59.927521 9079 consensus_queue.cc:237] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 13:59:59.928807 9205 raft_consensus.cc:1261] T 96db3ad20b864deb83dc3e2d64e2787e P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:59.929257 6659 raft_consensus.cc:1261] T 96db3ad20b864deb83dc3e2d64e2787e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:59.929664 16009 consensus_queue.cc:1035] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:59.930081 16010 consensus_queue.cc:1035] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:59.932559 13280 raft_consensus.cc:2932] T 96db3ad20b864deb83dc3e2d64e2787e P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:59.932933 6659 raft_consensus.cc:2932] T 96db3ad20b864deb83dc3e2d64e2787e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:59.934476 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 96db3ad20b864deb83dc3e2d64e2787e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:59.935094 9205 raft_consensus.cc:2932] T 96db3ad20b864deb83dc3e2d64e2787e P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:59.936301 6380 catalog_manager.cc:5484] T 96db3ad20b864deb83dc3e2d64e2787e P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index 3 to 4, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 13:59:59.939131 6769 tablet_service.cc:1514] Processing DeleteTablet for tablet 96db3ad20b864deb83dc3e2d64e2787e with delete_type TABLET_DATA_TOMBSTONED (TS 4a85e492c8154e9881a9cc2a105d7e64 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44858
I20240503 13:59:59.939371 15890 tablet_replica.cc:330] T 96db3ad20b864deb83dc3e2d64e2787e P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 13:59:59.939555 15890 raft_consensus.cc:2229] T 96db3ad20b864deb83dc3e2d64e2787e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:59.939857 15890 raft_consensus.cc:2258] T 96db3ad20b864deb83dc3e2d64e2787e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:59.940551 15890 ts_tablet_manager.cc:1875] T 96db3ad20b864deb83dc3e2d64e2787e P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:59.942819 15890 ts_tablet_manager.cc:1888] T 96db3ad20b864deb83dc3e2d64e2787e P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:59.942880 15890 log.cc:1196] T 96db3ad20b864deb83dc3e2d64e2787e P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/96db3ad20b864deb83dc3e2d64e2787e
I20240503 13:59:59.943658 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet 96db3ad20b864deb83dc3e2d64e2787e (table rebalancer_extra_table_60 [id=7a1d16a1055e477183d59c2aa87d96fb]) successfully deleted
I20240503 13:59:59.961771 9079 raft_consensus.cc:1203] T db14294a5205460dbdd0f7588d92c1b5 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:59.968554 11378 raft_consensus.cc:1050] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6: attempting to promote NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 to VOTER
I20240503 13:59:59.968978 11378 consensus_queue.cc:237] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 13:59:59.970229 6920 raft_consensus.cc:1261] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:59.970551 9079 raft_consensus.cc:1261] T 1be8960eed8148f5958f6bb412c30a7e P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:59.970551 6790 raft_consensus.cc:1261] T 1be8960eed8148f5958f6bb412c30a7e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:59.970912 14432 consensus_queue.cc:1035] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:59.971067 11378 consensus_queue.cc:1035] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:59.971177 15169 consensus_queue.cc:1035] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:59.973567 15169 raft_consensus.cc:2932] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:59.974849 9079 raft_consensus.cc:2932] T 1be8960eed8148f5958f6bb412c30a7e P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:59.976224 6790 raft_consensus.cc:2932] T 1be8960eed8148f5958f6bb412c30a7e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:59.976264 6921 raft_consensus.cc:2932] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:59.977123 6380 catalog_manager.cc:5484] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:59.982426 9342 consensus_queue.cc:237] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 13:59:59.983844 6921 raft_consensus.cc:1261] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:59.984320 15169 consensus_queue.cc:1035] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:59.984856 6790 raft_consensus.cc:1261] T 1be8960eed8148f5958f6bb412c30a7e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:59.985219 15169 consensus_queue.cc:1035] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:59.985957 11378 raft_consensus.cc:2932] T 1be8960eed8148f5958f6bb412c30a7e P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:59.986387 6921 raft_consensus.cc:2932] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:59.987243 6790 raft_consensus.cc:2932] T 1be8960eed8148f5958f6bb412c30a7e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:59.987596 6374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1be8960eed8148f5958f6bb412c30a7e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:59.989655 6385 catalog_manager.cc:5484] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 13:59:59.992578 9060 tablet_service.cc:1514] Processing DeleteTablet for tablet 1be8960eed8148f5958f6bb412c30a7e with delete_type TABLET_DATA_TOMBSTONED (TS 8d574fb4a89d4eea947c859470d44dd4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56018
I20240503 13:59:59.992808 15839 tablet_replica.cc:330] T 1be8960eed8148f5958f6bb412c30a7e P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 13:59:59.992978 15839 raft_consensus.cc:2229] T 1be8960eed8148f5958f6bb412c30a7e P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:59.993124 15839 raft_consensus.cc:2258] T 1be8960eed8148f5958f6bb412c30a7e P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:59.994076 15839 ts_tablet_manager.cc:1875] T 1be8960eed8148f5958f6bb412c30a7e P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:59.996275 15839 ts_tablet_manager.cc:1888] T 1be8960eed8148f5958f6bb412c30a7e P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 13:59:59.996348 15839 log.cc:1196] T 1be8960eed8148f5958f6bb412c30a7e P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/1be8960eed8148f5958f6bb412c30a7e
I20240503 13:59:59.997028 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet 1be8960eed8148f5958f6bb412c30a7e (table rebalancer_extra_table_44 [id=a8e55189edbc4c54bebc8aac90a27231]) successfully deleted
I20240503 14:00:00.009235 9342 raft_consensus.cc:1203] T a9e87083385d4e6bac9bda95670b68a8 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:00:00.032347 6921 consensus_queue.cc:237] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 14:00:00.034220 9080 raft_consensus.cc:1261] T 873f3615984646159e1b05178c933e18 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:00.034220 8947 raft_consensus.cc:1261] T 873f3615984646159e1b05178c933e18 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:00.034742 15111 consensus_queue.cc:1035] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:00:00.035125 15111 consensus_queue.cc:1035] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:00:00.036042 6858 consensus_peers.cc:485] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): Couldn't send request to peer 2145d15ab2d74c0287839266f1411a43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 873f3615984646159e1b05178c933e18. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:00.036885 15111 raft_consensus.cc:2932] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 14:00:00.039216 8947 raft_consensus.cc:2932] T 873f3615984646159e1b05178c933e18 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 14:00:00.039333 9080 raft_consensus.cc:2932] T 873f3615984646159e1b05178c933e18 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } } }
I20240503 14:00:00.042641 6387 catalog_manager.cc:5484] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index -1 to 2, NON_VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:00.054148 6790 consensus_queue.cc:237] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
W20240503 14:00:00.055835 6727 consensus_peers.cc:485] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 752e0d08505b4c8cb4b2a44cc00ce858. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:00.056003 6659 raft_consensus.cc:1261] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:00.056360 12032 consensus_queue.cc:1035] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:00.058013 12032 raft_consensus.cc:2932] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 14:00:00.058454 6659 raft_consensus.cc:2932] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 14:00:00.059928 6921 raft_consensus.cc:1261] T 752e0d08505b4c8cb4b2a44cc00ce858 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:00.060658 14879 consensus_queue.cc:1035] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:00.061683 6921 raft_consensus.cc:2932] T 752e0d08505b4c8cb4b2a44cc00ce858 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 14:00:00.062621 6385 catalog_manager.cc:5484] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:00.065837 12367 raft_consensus.cc:1050] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545: attempting to promote NON_VOTER 2145d15ab2d74c0287839266f1411a43 to VOTER
I20240503 14:00:00.066306 12367 consensus_queue.cc:237] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 14:00:00.067662 6527 raft_consensus.cc:1261] T 96d1a236cdb7406090021842d8dbc5a0 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:00:00.067801 9080 raft_consensus.cc:1261] T 96d1a236cdb7406090021842d8dbc5a0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:00:00.068092 14384 consensus_queue.cc:1035] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:00:00.068431 14384 consensus_queue.cc:1035] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:00:00.070376 6659 raft_consensus.cc:1261] T 96d1a236cdb7406090021842d8dbc5a0 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:00:00.071380 12367 consensus_queue.cc:1035] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:00:00.072340 12367 raft_consensus.cc:2932] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 14:00:00.073050 6920 consensus_queue.cc:237] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 14:00:00.073477 6790 raft_consensus.cc:1203] T 568c2e7206f9496baa22432b72c021b1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:00:00.074790 6658 raft_consensus.cc:1261] T 3856ff74b3bf4f608e3c6cdd06ce144e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:00.075260 15111 consensus_queue.cc:1035] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:00.076643 9080 raft_consensus.cc:2932] T 96d1a236cdb7406090021842d8dbc5a0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 14:00:00.077423 6387 catalog_manager.cc:5484] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:00.079200 6527 raft_consensus.cc:2932] T 96d1a236cdb7406090021842d8dbc5a0 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 14:00:00.080158 6658 raft_consensus.cc:2932] T 96d1a236cdb7406090021842d8dbc5a0 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 14:00:00.081548 6790 raft_consensus.cc:1261] T 3856ff74b3bf4f608e3c6cdd06ce144e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:00.082082 15111 consensus_queue.cc:1035] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:00:00.082513 6857 consensus_peers.cc:485] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): Couldn't send request to peer 06ca8dbc004c42b99faa527a4e6f9545. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3856ff74b3bf4f608e3c6cdd06ce144e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:00.086548 15111 raft_consensus.cc:2932] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 14:00:00.088742 6658 raft_consensus.cc:2932] T 3856ff74b3bf4f608e3c6cdd06ce144e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 14:00:00.089485 6790 raft_consensus.cc:2932] T 3856ff74b3bf4f608e3c6cdd06ce144e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 14:00:00.089910 8947 consensus_queue.cc:237] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 14:00:00.092293 6527 raft_consensus.cc:1261] T 96d1a236cdb7406090021842d8dbc5a0 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:00.092623 6658 raft_consensus.cc:1261] T 96d1a236cdb7406090021842d8dbc5a0 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:00.092880 14390 consensus_queue.cc:1035] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:00.093349 14390 consensus_queue.cc:1035] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:00.093365 6385 catalog_manager.cc:5484] T 3856ff74b3bf4f608e3c6cdd06ce144e P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 14:00:00.096550 15953 raft_consensus.cc:2932] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 14:00:00.098111 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 96d1a236cdb7406090021842d8dbc5a0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:00.099275 6527 raft_consensus.cc:2932] T 96d1a236cdb7406090021842d8dbc5a0 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 14:00:00.099522 6658 raft_consensus.cc:2932] T 96d1a236cdb7406090021842d8dbc5a0 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 14:00:00.103462 6387 catalog_manager.cc:5484] T 96d1a236cdb7406090021842d8dbc5a0 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:00.106156 9060 tablet_service.cc:1514] Processing DeleteTablet for tablet 96d1a236cdb7406090021842d8dbc5a0 with delete_type TABLET_DATA_TOMBSTONED (TS 8d574fb4a89d4eea947c859470d44dd4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56018
I20240503 14:00:00.106365 15839 tablet_replica.cc:330] T 96d1a236cdb7406090021842d8dbc5a0 P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 14:00:00.106514 15839 raft_consensus.cc:2229] T 96d1a236cdb7406090021842d8dbc5a0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:00.106626 15839 raft_consensus.cc:2258] T 96d1a236cdb7406090021842d8dbc5a0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:00.107836 15839 ts_tablet_manager.cc:1875] T 96d1a236cdb7406090021842d8dbc5a0 P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:00.109584 9079 consensus_queue.cc:237] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 14:00:00.110188 15839 ts_tablet_manager.cc:1888] T 96d1a236cdb7406090021842d8dbc5a0 P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:00:00.110255 15839 log.cc:1196] T 96d1a236cdb7406090021842d8dbc5a0 P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/96d1a236cdb7406090021842d8dbc5a0
I20240503 14:00:00.111546 6921 raft_consensus.cc:1261] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:00.111721 9205 raft_consensus.cc:1261] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:00.112010 15796 consensus_queue.cc:1035] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:00:00.112198 16010 consensus_queue.cc:1035] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:00:00.112563 15099 raft_consensus.cc:1050] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da: attempting to promote NON_VOTER 0d77752181e944698e809e325e5937e6 to VOTER
W20240503 14:00:00.112761 9017 consensus_peers.cc:485] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aaa93a7fd5df4cb7967d6bdb0c8dd6d1. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:00.112970 15099 consensus_queue.cc:237] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 14:00:00.113886 16010 raft_consensus.cc:2932] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 14:00:00.114228 9205 raft_consensus.cc:1261] T 9a6709e4024d47448c20d33a2b36479e P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:00:00.114801 15416 consensus_queue.cc:1035] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:00:00.115043 6658 raft_consensus.cc:1261] T 9a6709e4024d47448c20d33a2b36479e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:00:00.115624 9342 raft_consensus.cc:1261] T 9a6709e4024d47448c20d33a2b36479e P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:00:00.116320 15243 consensus_queue.cc:1035] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:00:00.116770 15243 consensus_queue.cc:1035] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:00:00.117144 9210 raft_consensus.cc:2932] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 14:00:00.117892 6380 catalog_manager.cc:5484] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index -1 to 2, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:00.121932 15416 raft_consensus.cc:2932] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 14:00:00.123890 9205 raft_consensus.cc:2932] T 9a6709e4024d47448c20d33a2b36479e P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 14:00:00.125067 6658 raft_consensus.cc:2932] T 9a6709e4024d47448c20d33a2b36479e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 14:00:00.127665 6920 raft_consensus.cc:2932] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 14:00:00.128288 6380 catalog_manager.cc:5484] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:00.133565 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet 96d1a236cdb7406090021842d8dbc5a0 (table rebalancer_extra_table_12 [id=3fa9362ac0d94323a9cd7a56209c3e07]) successfully deleted
I20240503 14:00:00.135176 9342 raft_consensus.cc:2932] T 9a6709e4024d47448c20d33a2b36479e P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 14:00:00.137975 6527 consensus_queue.cc:237] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 14:00:00.139627 6658 raft_consensus.cc:1261] T 9a6709e4024d47448c20d33a2b36479e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:00.140265 15972 consensus_queue.cc:1035] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:00.141490 9342 raft_consensus.cc:1261] T 9a6709e4024d47448c20d33a2b36479e P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:00.142061 15972 consensus_queue.cc:1035] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:00.143332 15972 raft_consensus.cc:2932] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 14:00:00.145123 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9a6709e4024d47448c20d33a2b36479e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:00.147182 6658 raft_consensus.cc:2932] T 9a6709e4024d47448c20d33a2b36479e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 14:00:00.147359 9342 raft_consensus.cc:2932] T 9a6709e4024d47448c20d33a2b36479e P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 14:00:00.147913 6385 catalog_manager.cc:5484] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:00.151161 9192 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a6709e4024d47448c20d33a2b36479e with delete_type TABLET_DATA_TOMBSTONED (TS ab1d21935d6049888eb0ceccc3c9d735 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57960
I20240503 14:00:00.154237 16115 tablet_replica.cc:330] T 9a6709e4024d47448c20d33a2b36479e P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 14:00:00.154457 16115 raft_consensus.cc:2229] T 9a6709e4024d47448c20d33a2b36479e P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:00.154672 16115 raft_consensus.cc:2258] T 9a6709e4024d47448c20d33a2b36479e P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:00.155514 16115 ts_tablet_manager.cc:1875] T 9a6709e4024d47448c20d33a2b36479e P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:00.158285 16115 ts_tablet_manager.cc:1888] T 9a6709e4024d47448c20d33a2b36479e P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:00:00.158491 16115 log.cc:1196] T 9a6709e4024d47448c20d33a2b36479e P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/9a6709e4024d47448c20d33a2b36479e
I20240503 14:00:00.159194 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet 9a6709e4024d47448c20d33a2b36479e (table rebalancer_extra_table_48 [id=5acb9e8cbce6439bbadafaf4632200c2]) successfully deleted
I20240503 14:00:00.246054 16186 ts_tablet_manager.cc:902] T d51634c08e2749d4a3bbff4b2a786e48 P ab1d21935d6049888eb0ceccc3c9d735: Initiating tablet copy from peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 14:00:00.246698 16186 tablet_copy_client.cc:308] T d51634c08e2749d4a3bbff4b2a786e48 P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:34547
I20240503 14:00:00.247139 8968 tablet_copy_service.cc:140] P 06ca8dbc004c42b99faa527a4e6f9545: Received BeginTabletCopySession request for tablet d51634c08e2749d4a3bbff4b2a786e48 from peer ab1d21935d6049888eb0ceccc3c9d735 ({username='slave'} at 127.0.99.135:33143)
I20240503 14:00:00.247241 8968 tablet_copy_service.cc:161] P 06ca8dbc004c42b99faa527a4e6f9545: Beginning new tablet copy session on tablet d51634c08e2749d4a3bbff4b2a786e48 from peer ab1d21935d6049888eb0ceccc3c9d735 at {username='slave'} at 127.0.99.135:33143: session id = ab1d21935d6049888eb0ceccc3c9d735-d51634c08e2749d4a3bbff4b2a786e48
I20240503 14:00:00.248347 8968 tablet_copy_source_session.cc:205] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:00.249657 16186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d51634c08e2749d4a3bbff4b2a786e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:00.252458 16186 tablet_copy_client.cc:783] T d51634c08e2749d4a3bbff4b2a786e48 P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:00.252698 16186 tablet_copy_client.cc:647] T d51634c08e2749d4a3bbff4b2a786e48 P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:00.253844 16186 tablet_copy_client.cc:523] T d51634c08e2749d4a3bbff4b2a786e48 P ab1d21935d6049888eb0ceccc3c9d735: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:00.255170 16186 tablet_bootstrap.cc:492] T d51634c08e2749d4a3bbff4b2a786e48 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap starting.
I20240503 14:00:00.262496 16186 tablet_bootstrap.cc:492] T d51634c08e2749d4a3bbff4b2a786e48 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:00.262807 16186 tablet_bootstrap.cc:492] T d51634c08e2749d4a3bbff4b2a786e48 P ab1d21935d6049888eb0ceccc3c9d735: Bootstrap complete.
I20240503 14:00:00.262985 16186 ts_tablet_manager.cc:1367] T d51634c08e2749d4a3bbff4b2a786e48 P ab1d21935d6049888eb0ceccc3c9d735: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240503 14:00:00.263520 16186 raft_consensus.cc:348] T d51634c08e2749d4a3bbff4b2a786e48 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 14:00:00.263643 16186 raft_consensus.cc:729] T d51634c08e2749d4a3bbff4b2a786e48 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Initialized, Role: LEARNER
I20240503 14:00:00.263742 16186 consensus_queue.cc:260] T d51634c08e2749d4a3bbff4b2a786e48 P ab1d21935d6049888eb0ceccc3c9d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: true } }
I20240503 14:00:00.264115 16186 ts_tablet_manager.cc:1398] T d51634c08e2749d4a3bbff4b2a786e48 P ab1d21935d6049888eb0ceccc3c9d735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:00.264758 8968 tablet_copy_service.cc:342] P 06ca8dbc004c42b99faa527a4e6f9545: Request end of tablet copy session ab1d21935d6049888eb0ceccc3c9d735-d51634c08e2749d4a3bbff4b2a786e48 received from {username='slave'} at 127.0.99.135:33143
I20240503 14:00:00.264837 8968 tablet_copy_service.cc:433] P 06ca8dbc004c42b99faa527a4e6f9545: ending tablet copy session ab1d21935d6049888eb0ceccc3c9d735-d51634c08e2749d4a3bbff4b2a786e48 on tablet d51634c08e2749d4a3bbff4b2a786e48 with peer ab1d21935d6049888eb0ceccc3c9d735
I20240503 14:00:00.265769 6659 consensus_queue.cc:237] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 14:00:00.267371 6790 raft_consensus.cc:1261] T 21dd1c7418e54d30b665f57ef1842f9e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:00:00.267644 6596 consensus_peers.cc:485] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21dd1c7418e54d30b665f57ef1842f9e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:00.268069 6920 raft_consensus.cc:1261] T 21dd1c7418e54d30b665f57ef1842f9e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:00.268550 14177 consensus_queue.cc:1035] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:00.268745 15876 consensus_queue.cc:1035] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:00.270721 14177 raft_consensus.cc:2932] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 14:00:00.271066 6920 raft_consensus.cc:2932] T 21dd1c7418e54d30b665f57ef1842f9e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 14:00:00.271468 6790 raft_consensus.cc:2932] T 21dd1c7418e54d30b665f57ef1842f9e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } } }
I20240503 14:00:00.274487 6385 catalog_manager.cc:5484] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index -1 to 4, NON_VOTER 0d77752181e944698e809e325e5937e6 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:00.283974 8946 consensus_queue.cc:237] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 14:00:00.285260 9080 raft_consensus.cc:1261] T 9af92a2d926543c4aa68a1f85b7da24b P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:00:00.285362 8882 consensus_peers.cc:485] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9af92a2d926543c4aa68a1f85b7da24b. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:00.285393 9342 raft_consensus.cc:1261] T 9af92a2d926543c4aa68a1f85b7da24b P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:00.285714 14390 consensus_queue.cc:1035] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:00:00.285883 14384 consensus_queue.cc:1035] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:00:00.287415 15432 raft_consensus.cc:1050] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43: attempting to promote NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 to VOTER
I20240503 14:00:00.287732 14390 raft_consensus.cc:2932] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 14:00:00.287829 15432 consensus_queue.cc:237] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
I20240503 14:00:00.288236 9080 raft_consensus.cc:2932] T 9af92a2d926543c4aa68a1f85b7da24b P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 14:00:00.289624 6790 raft_consensus.cc:1261] T 9de4c2ac66344f719b3fe56ffbcc804d P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:00.290163 15959 consensus_queue.cc:1035] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:00.290328 6526 raft_consensus.cc:1261] T 9de4c2ac66344f719b3fe56ffbcc804d P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:00.290717 15959 consensus_queue.cc:1035] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:00.290820 9342 raft_consensus.cc:2932] T 9af92a2d926543c4aa68a1f85b7da24b P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 14:00:00.292150 8946 raft_consensus.cc:1261] T 9de4c2ac66344f719b3fe56ffbcc804d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:00.293483 6380 catalog_manager.cc:5484] T 9af92a2d926543c4aa68a1f85b7da24b P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 14:00:00.295343 15959 consensus_queue.cc:1035] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:00.297240 15101 raft_consensus.cc:2932] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 14:00:00.298867 8946 raft_consensus.cc:2932] T 9de4c2ac66344f719b3fe56ffbcc804d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 14:00:00.299849 6526 raft_consensus.cc:2932] T 9de4c2ac66344f719b3fe56ffbcc804d P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 14:00:00.302214 6790 raft_consensus.cc:2932] T 9de4c2ac66344f719b3fe56ffbcc804d P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 14:00:00.303682 6387 catalog_manager.cc:5484] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:00.309698 6659 consensus_queue.cc:237] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
I20240503 14:00:00.310741 8947 raft_consensus.cc:1261] T 9de4c2ac66344f719b3fe56ffbcc804d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:00.311163 6526 raft_consensus.cc:1261] T 9de4c2ac66344f719b3fe56ffbcc804d P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:00.311170 15432 consensus_queue.cc:1035] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:00.311585 15432 consensus_queue.cc:1035] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:00.312984 15959 raft_consensus.cc:2932] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 14:00:00.313417 8947 raft_consensus.cc:2932] T 9de4c2ac66344f719b3fe56ffbcc804d P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 14:00:00.314553 6374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9de4c2ac66344f719b3fe56ffbcc804d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:00.317014 6380 catalog_manager.cc:5484] T 9de4c2ac66344f719b3fe56ffbcc804d P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index 5 to 6, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:00.317545 6526 raft_consensus.cc:2932] T 9de4c2ac66344f719b3fe56ffbcc804d P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 14:00:00.319430 6769 tablet_service.cc:1514] Processing DeleteTablet for tablet 9de4c2ac66344f719b3fe56ffbcc804d with delete_type TABLET_DATA_TOMBSTONED (TS 4a85e492c8154e9881a9cc2a105d7e64 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44858
I20240503 14:00:00.319686 15890 tablet_replica.cc:330] T 9de4c2ac66344f719b3fe56ffbcc804d P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 14:00:00.319802 15890 raft_consensus.cc:2229] T 9de4c2ac66344f719b3fe56ffbcc804d P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:00.319947 15890 raft_consensus.cc:2258] T 9de4c2ac66344f719b3fe56ffbcc804d P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:00.320694 15890 ts_tablet_manager.cc:1875] T 9de4c2ac66344f719b3fe56ffbcc804d P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:00.322651 15890 ts_tablet_manager.cc:1888] T 9de4c2ac66344f719b3fe56ffbcc804d P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:00:00.322731 15890 log.cc:1196] T 9de4c2ac66344f719b3fe56ffbcc804d P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/9de4c2ac66344f719b3fe56ffbcc804d
I20240503 14:00:00.323194 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet 9de4c2ac66344f719b3fe56ffbcc804d (table rebalance_test_table_1 [id=f757e59886674aafb5f975c30459df7e]) successfully deleted
I20240503 14:00:00.324045 14879 raft_consensus.cc:1050] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64: attempting to promote NON_VOTER f58065db70b64ef8b1dc499aa6ae4460 to VOTER
I20240503 14:00:00.324450 14879 consensus_queue.cc:237] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 14:00:00.325656 6659 raft_consensus.cc:1261] T ea438ff21d9744f697878406493f0fdc P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:00:00.325649 6526 raft_consensus.cc:1261] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:00:00.325964 14879 consensus_queue.cc:1035] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:00:00.326150 6920 raft_consensus.cc:1261] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEARNER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:00:00.326320 14879 consensus_queue.cc:1035] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:00:00.327229 14879 consensus_queue.cc:1035] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:00:00.328660 14879 raft_consensus.cc:2932] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 14:00:00.328954 6526 raft_consensus.cc:2932] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 14:00:00.332587 6379 catalog_manager.cc:5484] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 14:00:00.333143 6659 raft_consensus.cc:2932] T ea438ff21d9744f697878406493f0fdc P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 14:00:00.333169 6920 raft_consensus.cc:2932] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 14:00:00.340461 6790 consensus_queue.cc:237] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 14:00:00.341372 16232 ts_tablet_manager.cc:902] T b6cb5119a6db403ba3cae9fa9b121211 P 06ca8dbc004c42b99faa527a4e6f9545: Initiating tablet copy from peer 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 14:00:00.341974 16232 tablet_copy_client.cc:308] T b6cb5119a6db403ba3cae9fa9b121211 P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46119
I20240503 14:00:00.342388 6920 raft_consensus.cc:1261] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:00.342733 6526 raft_consensus.cc:1261] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:00.343139 6680 tablet_copy_service.cc:140] P 2145d15ab2d74c0287839266f1411a43: Received BeginTabletCopySession request for tablet b6cb5119a6db403ba3cae9fa9b121211 from peer 06ca8dbc004c42b99faa527a4e6f9545 ({username='slave'} at 127.0.99.133:36361)
I20240503 14:00:00.343225 6680 tablet_copy_service.cc:161] P 2145d15ab2d74c0287839266f1411a43: Beginning new tablet copy session on tablet b6cb5119a6db403ba3cae9fa9b121211 from peer 06ca8dbc004c42b99faa527a4e6f9545 at {username='slave'} at 127.0.99.133:36361: session id = 06ca8dbc004c42b99faa527a4e6f9545-b6cb5119a6db403ba3cae9fa9b121211
I20240503 14:00:00.343237 15149 consensus_queue.cc:1035] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:00.343394 16223 consensus_queue.cc:1035] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:00.344215 6680 tablet_copy_source_session.cc:205] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:00.345470 16232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6cb5119a6db403ba3cae9fa9b121211. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:00.345892 14879 raft_consensus.cc:2932] T ea438ff21d9744f697878406493f0fdc P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 14:00:00.347244 6372 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ea438ff21d9744f697878406493f0fdc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:00.347908 16232 tablet_copy_client.cc:783] T b6cb5119a6db403ba3cae9fa9b121211 P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:00.348109 16232 tablet_copy_client.cc:647] T b6cb5119a6db403ba3cae9fa9b121211 P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:00.347997 6920 raft_consensus.cc:2932] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 14:00:00.349475 16232 tablet_copy_client.cc:523] T b6cb5119a6db403ba3cae9fa9b121211 P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:00.351133 16232 tablet_bootstrap.cc:492] T b6cb5119a6db403ba3cae9fa9b121211 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 14:00:00.351528 6379 catalog_manager.cc:5484] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 3 to 4, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 14:00:00.351827 9342 raft_consensus.cc:1203] T d55d270456d3460586ff3cd41787b5d2 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:00.352517 6527 raft_consensus.cc:2932] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } } }
I20240503 14:00:00.356598 6639 tablet_service.cc:1514] Processing DeleteTablet for tablet ea438ff21d9744f697878406493f0fdc with delete_type TABLET_DATA_TOMBSTONED (TS 2145d15ab2d74c0287839266f1411a43 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37976
I20240503 14:00:00.360464 16242 tablet_replica.cc:330] T ea438ff21d9744f697878406493f0fdc P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 14:00:00.360639 16242 raft_consensus.cc:2229] T ea438ff21d9744f697878406493f0fdc P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:00.360805 16242 raft_consensus.cc:2258] T ea438ff21d9744f697878406493f0fdc P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:00.364926 16242 ts_tablet_manager.cc:1875] T ea438ff21d9744f697878406493f0fdc P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:00.373769 16242 ts_tablet_manager.cc:1888] T ea438ff21d9744f697878406493f0fdc P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:00:00.373857 16242 log.cc:1196] T ea438ff21d9744f697878406493f0fdc P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/ea438ff21d9744f697878406493f0fdc
I20240503 14:00:00.374682 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet ea438ff21d9744f697878406493f0fdc (table rebalancer_extra_table_60 [id=7a1d16a1055e477183d59c2aa87d96fb]) successfully deleted
I20240503 14:00:00.388777 16232 tablet_bootstrap.cc:492] T b6cb5119a6db403ba3cae9fa9b121211 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:00.389235 16232 tablet_bootstrap.cc:492] T b6cb5119a6db403ba3cae9fa9b121211 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap complete.
I20240503 14:00:00.389442 16232 ts_tablet_manager.cc:1367] T b6cb5119a6db403ba3cae9fa9b121211 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.038s user 0.025s sys 0.000s
I20240503 14:00:00.389892 16232 raft_consensus.cc:348] T b6cb5119a6db403ba3cae9fa9b121211 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 14:00:00.390002 16232 raft_consensus.cc:729] T b6cb5119a6db403ba3cae9fa9b121211 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: LEARNER
I20240503 14:00:00.390084 16232 consensus_queue.cc:260] T b6cb5119a6db403ba3cae9fa9b121211 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 14:00:00.391243 16232 ts_tablet_manager.cc:1398] T b6cb5119a6db403ba3cae9fa9b121211 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:00.391749 6680 tablet_copy_service.cc:342] P 2145d15ab2d74c0287839266f1411a43: Request end of tablet copy session 06ca8dbc004c42b99faa527a4e6f9545-b6cb5119a6db403ba3cae9fa9b121211 received from {username='slave'} at 127.0.99.133:36361
I20240503 14:00:00.391834 6680 tablet_copy_service.cc:433] P 2145d15ab2d74c0287839266f1411a43: ending tablet copy session 06ca8dbc004c42b99faa527a4e6f9545-b6cb5119a6db403ba3cae9fa9b121211 on tablet b6cb5119a6db403ba3cae9fa9b121211 with peer 06ca8dbc004c42b99faa527a4e6f9545
I20240503 14:00:00.453421 16096 raft_consensus.cc:1050] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735: attempting to promote NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 to VOTER
I20240503 14:00:00.453886 16096 consensus_queue.cc:237] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 14:00:00.455334 8947 raft_consensus.cc:1261] T 568c2e7206f9496baa22432b72c021b1 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:00:00.455440 9079 raft_consensus.cc:1261] T 568c2e7206f9496baa22432b72c021b1 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:00:00.455647 6790 raft_consensus.cc:1261] T 568c2e7206f9496baa22432b72c021b1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:00:00.455804 16309 consensus_queue.cc:1035] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:00:00.455999 12063 consensus_queue.cc:1035] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:00:00.456135 14486 consensus_queue.cc:1035] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:00:00.456214 16310 ts_tablet_manager.cc:902] T 752e0d08505b4c8cb4b2a44cc00ce858 P 0d77752181e944698e809e325e5937e6: Initiating tablet copy from peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 14:00:00.456686 16310 tablet_copy_client.cc:308] T 752e0d08505b4c8cb4b2a44cc00ce858 P 0d77752181e944698e809e325e5937e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:46365
I20240503 14:00:00.457410 6811 tablet_copy_service.cc:140] P 4a85e492c8154e9881a9cc2a105d7e64: Received BeginTabletCopySession request for tablet 752e0d08505b4c8cb4b2a44cc00ce858 from peer 0d77752181e944698e809e325e5937e6 ({username='slave'} at 127.0.99.136:34769)
I20240503 14:00:00.457494 6811 tablet_copy_service.cc:161] P 4a85e492c8154e9881a9cc2a105d7e64: Beginning new tablet copy session on tablet 752e0d08505b4c8cb4b2a44cc00ce858 from peer 0d77752181e944698e809e325e5937e6 at {username='slave'} at 127.0.99.136:34769: session id = 0d77752181e944698e809e325e5937e6-752e0d08505b4c8cb4b2a44cc00ce858
I20240503 14:00:00.458447 6811 tablet_copy_source_session.cc:205] T 752e0d08505b4c8cb4b2a44cc00ce858 P 4a85e492c8154e9881a9cc2a105d7e64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:00.459189 12063 raft_consensus.cc:2932] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 14:00:00.459357 16310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752e0d08505b4c8cb4b2a44cc00ce858. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:00.460975 9079 raft_consensus.cc:2932] T 568c2e7206f9496baa22432b72c021b1 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 14:00:00.461535 16310 tablet_copy_client.cc:783] T 752e0d08505b4c8cb4b2a44cc00ce858 P 0d77752181e944698e809e325e5937e6: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:00.461668 16310 tablet_copy_client.cc:647] T 752e0d08505b4c8cb4b2a44cc00ce858 P 0d77752181e944698e809e325e5937e6: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:00.462491 6790 raft_consensus.cc:2932] T 568c2e7206f9496baa22432b72c021b1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 14:00:00.462978 6385 catalog_manager.cc:5484] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index 2 to 3, 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:00.463697 16310 tablet_copy_client.cc:523] T 752e0d08505b4c8cb4b2a44cc00ce858 P 0d77752181e944698e809e325e5937e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:00.465390 16310 tablet_bootstrap.cc:492] T 752e0d08505b4c8cb4b2a44cc00ce858 P 0d77752181e944698e809e325e5937e6: Bootstrap starting.
I20240503 14:00:00.466660 9205 consensus_queue.cc:237] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 14:00:00.468055 6790 raft_consensus.cc:1261] T 568c2e7206f9496baa22432b72c021b1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:00.468556 16096 consensus_queue.cc:1035] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:00.471110 14486 raft_consensus.cc:2932] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 14:00:00.472297 9079 raft_consensus.cc:1261] T 568c2e7206f9496baa22432b72c021b1 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:00.474218 12063 consensus_queue.cc:1035] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:00.474519 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 568c2e7206f9496baa22432b72c021b1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:00.475059 6788 raft_consensus.cc:2932] T 568c2e7206f9496baa22432b72c021b1 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 14:00:00.475739 9080 raft_consensus.cc:2932] T 568c2e7206f9496baa22432b72c021b1 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } } }
I20240503 14:00:00.476254 14432 raft_consensus.cc:1050] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6: attempting to promote NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 to VOTER
I20240503 14:00:00.476620 14432 consensus_queue.cc:237] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 14:00:00.478096 9205 raft_consensus.cc:1261] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:00:00.478799 6920 raft_consensus.cc:1261] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:00:00.478787 15169 consensus_queue.cc:1035] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:00:00.478873 8947 raft_consensus.cc:1261] T a914c22ceff44fc685aad0fdcd04d844 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:00:00.479451 11378 consensus_queue.cc:1035] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:00:00.479749 11378 consensus_queue.cc:1035] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:00:00.483758 6380 catalog_manager.cc:5484] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:00.483779 15169 raft_consensus.cc:2932] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 14:00:00.486460 6788 consensus_queue.cc:237] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 14:00:00.488001 6921 raft_consensus.cc:1261] T 586369c0c3174070a3e184af38b3adf7 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:00:00.488281 6726 consensus_peers.cc:485] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64 -> Peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): Couldn't send request to peer 06ca8dbc004c42b99faa527a4e6f9545. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 586369c0c3174070a3e184af38b3adf7. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:00.488327 12748 raft_consensus.cc:1050] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460: attempting to promote NON_VOTER 8d574fb4a89d4eea947c859470d44dd4 to VOTER
I20240503 14:00:00.488476 12032 consensus_queue.cc:1035] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:00.488714 12748 consensus_queue.cc:237] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 14:00:00.489835 6387 catalog_manager.cc:5484] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 reported cstate change: config changed from index 2 to 3, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:00.490021 6659 raft_consensus.cc:1261] T db14294a5205460dbdd0f7588d92c1b5 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:00.490114 9080 raft_consensus.cc:1261] T db14294a5205460dbdd0f7588d92c1b5 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 LEARNER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:00.490399 12748 consensus_queue.cc:1035] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:00.490556 13370 consensus_queue.cc:1035] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:00.491766 6789 raft_consensus.cc:1261] T db14294a5205460dbdd0f7588d92c1b5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:00.492725 12748 consensus_queue.cc:1035] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:00.492970 6658 raft_consensus.cc:1261] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:00.493321 12032 consensus_queue.cc:1035] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:00.494489 16232 ts_tablet_manager.cc:902] T 3856ff74b3bf4f608e3c6cdd06ce144e P 06ca8dbc004c42b99faa527a4e6f9545: Initiating tablet copy from peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 14:00:00.494954 16232 tablet_copy_client.cc:308] T 3856ff74b3bf4f608e3c6cdd06ce144e P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43531
I20240503 14:00:00.495585 6942 tablet_copy_service.cc:140] P f58065db70b64ef8b1dc499aa6ae4460: Received BeginTabletCopySession request for tablet 3856ff74b3bf4f608e3c6cdd06ce144e from peer 06ca8dbc004c42b99faa527a4e6f9545 ({username='slave'} at 127.0.99.133:60451)
I20240503 14:00:00.495697 6942 tablet_copy_service.cc:161] P f58065db70b64ef8b1dc499aa6ae4460: Beginning new tablet copy session on tablet 3856ff74b3bf4f608e3c6cdd06ce144e from peer 06ca8dbc004c42b99faa527a4e6f9545 at {username='slave'} at 127.0.99.133:60451: session id = 06ca8dbc004c42b99faa527a4e6f9545-3856ff74b3bf4f608e3c6cdd06ce144e
I20240503 14:00:00.495594 8947 raft_consensus.cc:2932] T a914c22ceff44fc685aad0fdcd04d844 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 14:00:00.495649 9205 raft_consensus.cc:2932] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 14:00:00.496659 6942 tablet_copy_source_session.cc:205] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:00.498447 16232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3856ff74b3bf4f608e3c6cdd06ce144e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:00.499116 8926 tablet_service.cc:1514] Processing DeleteTablet for tablet 568c2e7206f9496baa22432b72c021b1 with delete_type TABLET_DATA_TOMBSTONED (TS 06ca8dbc004c42b99faa527a4e6f9545 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38372
I20240503 14:00:00.500631 16335 tablet_replica.cc:330] T 568c2e7206f9496baa22432b72c021b1 P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 14:00:00.500852 16335 raft_consensus.cc:2229] T 568c2e7206f9496baa22432b72c021b1 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:00.501083 16335 raft_consensus.cc:2258] T 568c2e7206f9496baa22432b72c021b1 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:00.501681 16335 ts_tablet_manager.cc:1875] T 568c2e7206f9496baa22432b72c021b1 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:00.501834 16232 tablet_copy_client.cc:783] T 3856ff74b3bf4f608e3c6cdd06ce144e P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:00.501989 16232 tablet_copy_client.cc:647] T 3856ff74b3bf4f608e3c6cdd06ce144e P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:00.502509 12032 raft_consensus.cc:2932] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 14:00:00.504703 6921 raft_consensus.cc:2932] T 586369c0c3174070a3e184af38b3adf7 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 14:00:00.506021 6658 raft_consensus.cc:2932] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 14:00:00.506871 6387 catalog_manager.cc:5484] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:00.508051 13370 raft_consensus.cc:2932] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 14:00:00.511101 9080 raft_consensus.cc:2932] T db14294a5205460dbdd0f7588d92c1b5 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 14:00:00.514199 6658 raft_consensus.cc:2932] T db14294a5205460dbdd0f7588d92c1b5 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 14:00:00.516360 6920 raft_consensus.cc:2932] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 14:00:00.518265 9342 consensus_queue.cc:237] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 14:00:00.519465 9205 raft_consensus.cc:1261] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:00.519955 6380 catalog_manager.cc:5484] T db14294a5205460dbdd0f7588d92c1b5 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index 4 to 5, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 14:00:00.520849 16232 tablet_copy_client.cc:523] T 3856ff74b3bf4f608e3c6cdd06ce144e P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:00.520694 6789 raft_consensus.cc:2932] T db14294a5205460dbdd0f7588d92c1b5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 14:00:00.522439 16232 tablet_bootstrap.cc:492] T 3856ff74b3bf4f608e3c6cdd06ce144e P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 14:00:00.522753 11378 consensus_queue.cc:1035] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:00.526849 6920 raft_consensus.cc:1261] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 0d77752181e944698e809e325e5937e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:00.529520 11378 consensus_queue.cc:1035] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:00.532182 11378 raft_consensus.cc:2932] T a914c22ceff44fc685aad0fdcd04d844 P 0d77752181e944698e809e325e5937e6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 14:00:00.532928 6922 consensus_queue.cc:237] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 14:00:00.534380 9080 raft_consensus.cc:1261] T db14294a5205460dbdd0f7588d92c1b5 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:00.534761 12748 consensus_queue.cc:1035] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:00.535166 6374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a914c22ceff44fc685aad0fdcd04d844 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:00.535847 9205 raft_consensus.cc:2932] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 14:00:00.538043 6922 raft_consensus.cc:2932] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 14:00:00.538642 12748 raft_consensus.cc:2932] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 14:00:00.540670 6385 catalog_manager.cc:5484] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index 3 to 4, VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "0d77752181e944698e809e325e5937e6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 14:00:00.542053 9080 raft_consensus.cc:2932] T db14294a5205460dbdd0f7588d92c1b5 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 14:00:00.542814 6658 raft_consensus.cc:1261] T db14294a5205460dbdd0f7588d92c1b5 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:00.543359 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet db14294a5205460dbdd0f7588d92c1b5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:00.544683 13370 consensus_queue.cc:1035] T db14294a5205460dbdd0f7588d92c1b5 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:00.546335 6379 catalog_manager.cc:5484] T db14294a5205460dbdd0f7588d92c1b5 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index 5 to 6, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 14:00:00.546934 8926 tablet_service.cc:1514] Processing DeleteTablet for tablet a914c22ceff44fc685aad0fdcd04d844 with delete_type TABLET_DATA_TOMBSTONED (TS 06ca8dbc004c42b99faa527a4e6f9545 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38372
I20240503 14:00:00.548008 6659 raft_consensus.cc:2932] T db14294a5205460dbdd0f7588d92c1b5 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } } }
I20240503 14:00:00.550963 6769 tablet_service.cc:1514] Processing DeleteTablet for tablet db14294a5205460dbdd0f7588d92c1b5 with delete_type TABLET_DATA_TOMBSTONED (TS 4a85e492c8154e9881a9cc2a105d7e64 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44858
I20240503 14:00:00.551167 15890 tablet_replica.cc:330] T db14294a5205460dbdd0f7588d92c1b5 P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 14:00:00.551321 15890 raft_consensus.cc:2229] T db14294a5205460dbdd0f7588d92c1b5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:00.551463 15890 raft_consensus.cc:2258] T db14294a5205460dbdd0f7588d92c1b5 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:00.552197 15890 ts_tablet_manager.cc:1875] T db14294a5205460dbdd0f7588d92c1b5 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:00.554594 15890 ts_tablet_manager.cc:1888] T db14294a5205460dbdd0f7588d92c1b5 P 4a85e492c8154e9881a9cc2a105d7e64: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:00:00.554662 15890 log.cc:1196] T db14294a5205460dbdd0f7588d92c1b5 P 4a85e492c8154e9881a9cc2a105d7e64: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/db14294a5205460dbdd0f7588d92c1b5
I20240503 14:00:00.555143 6372 catalog_manager.cc:4830] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): tablet db14294a5205460dbdd0f7588d92c1b5 (table rebalance_test_table_7 [id=73b39055500b4618821faa094d33670a]) successfully deleted
I20240503 14:00:00.559621 16335 ts_tablet_manager.cc:1888] T 568c2e7206f9496baa22432b72c021b1 P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:00:00.559614 6659 consensus_queue.cc:237] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 14:00:00.559988 16335 log.cc:1196] T 568c2e7206f9496baa22432b72c021b1 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/568c2e7206f9496baa22432b72c021b1
I20240503 14:00:00.561429 9205 raft_consensus.cc:1261] T 564582aa33a7465081724523096591d3 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:00:00.561684 6594 consensus_peers.cc:485] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 564582aa33a7465081724523096591d3. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:00.562021 16342 ts_tablet_manager.cc:902] T 873f3615984646159e1b05178c933e18 P 2145d15ab2d74c0287839266f1411a43: Initiating tablet copy from peer f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 14:00:00.561978 15101 consensus_queue.cc:1035] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:00:00.563004 16342 tablet_copy_client.cc:308] T 873f3615984646159e1b05178c933e18 P 2145d15ab2d74c0287839266f1411a43: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43531
I20240503 14:00:00.563261 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet 568c2e7206f9496baa22432b72c021b1 (table rebalancer_extra_table_6 [id=48b6af18e5404209abb657e70e9adedb]) successfully deleted
I20240503 14:00:00.563608 6942 tablet_copy_service.cc:140] P f58065db70b64ef8b1dc499aa6ae4460: Received BeginTabletCopySession request for tablet 873f3615984646159e1b05178c933e18 from peer 2145d15ab2d74c0287839266f1411a43 ({username='slave'} at 127.0.99.130:57151)
I20240503 14:00:00.563666 16335 tablet_replica.cc:330] T a914c22ceff44fc685aad0fdcd04d844 P 06ca8dbc004c42b99faa527a4e6f9545: stopping tablet replica
I20240503 14:00:00.563709 6942 tablet_copy_service.cc:161] P f58065db70b64ef8b1dc499aa6ae4460: Beginning new tablet copy session on tablet 873f3615984646159e1b05178c933e18 from peer 2145d15ab2d74c0287839266f1411a43 at {username='slave'} at 127.0.99.130:57151: session id = 2145d15ab2d74c0287839266f1411a43-873f3615984646159e1b05178c933e18
I20240503 14:00:00.563803 16335 raft_consensus.cc:2229] T a914c22ceff44fc685aad0fdcd04d844 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:00.563807 9080 raft_consensus.cc:1261] T 564582aa33a7465081724523096591d3 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:00.563946 16335 raft_consensus.cc:2258] T a914c22ceff44fc685aad0fdcd04d844 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:00.564729 6942 tablet_copy_source_session.cc:205] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:00.564896 15432 consensus_queue.cc:1035] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:00:00.565001 16335 ts_tablet_manager.cc:1875] T a914c22ceff44fc685aad0fdcd04d844 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:00.567106 15432 raft_consensus.cc:2932] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 14:00:00.571552 16342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 873f3615984646159e1b05178c933e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:00.575304 9205 raft_consensus.cc:2932] T 564582aa33a7465081724523096591d3 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 14:00:00.575855 16342 tablet_copy_client.cc:783] T 873f3615984646159e1b05178c933e18 P 2145d15ab2d74c0287839266f1411a43: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:00.575951 16310 tablet_bootstrap.cc:492] T 752e0d08505b4c8cb4b2a44cc00ce858 P 0d77752181e944698e809e325e5937e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:00.576004 16342 tablet_copy_client.cc:647] T 873f3615984646159e1b05178c933e18 P 2145d15ab2d74c0287839266f1411a43: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:00.576377 16310 tablet_bootstrap.cc:492] T 752e0d08505b4c8cb4b2a44cc00ce858 P 0d77752181e944698e809e325e5937e6: Bootstrap complete.
I20240503 14:00:00.577332 9081 raft_consensus.cc:2932] T 564582aa33a7465081724523096591d3 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 14:00:00.577760 16335 ts_tablet_manager.cc:1888] T a914c22ceff44fc685aad0fdcd04d844 P 06ca8dbc004c42b99faa527a4e6f9545: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:00:00.577823 16335 log.cc:1196] T a914c22ceff44fc685aad0fdcd04d844 P 06ca8dbc004c42b99faa527a4e6f9545: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/a914c22ceff44fc685aad0fdcd04d844
I20240503 14:00:00.578066 16310 ts_tablet_manager.cc:1367] T 752e0d08505b4c8cb4b2a44cc00ce858 P 0d77752181e944698e809e325e5937e6: Time spent bootstrapping tablet: real 0.113s user 0.025s sys 0.005s
I20240503 14:00:00.578361 6373 catalog_manager.cc:4830] TS 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): tablet a914c22ceff44fc685aad0fdcd04d844 (table rebalancer_extra_table_46 [id=8e3c65bd2d8c4b5595ccf6d282f48a82]) successfully deleted
I20240503 14:00:00.578585 16310 raft_consensus.cc:348] T 752e0d08505b4c8cb4b2a44cc00ce858 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 14:00:00.578850 16310 raft_consensus.cc:729] T 752e0d08505b4c8cb4b2a44cc00ce858 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0d77752181e944698e809e325e5937e6, State: Initialized, Role: LEARNER
I20240503 14:00:00.579051 16310 consensus_queue.cc:260] T 752e0d08505b4c8cb4b2a44cc00ce858 P 0d77752181e944698e809e325e5937e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 14:00:00.580325 6380 catalog_manager.cc:5484] T 564582aa33a7465081724523096591d3 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 14:00:00.580983 16310 ts_tablet_manager.cc:1398] T 752e0d08505b4c8cb4b2a44cc00ce858 P 0d77752181e944698e809e325e5937e6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:00:00.582561 6811 tablet_copy_service.cc:342] P 4a85e492c8154e9881a9cc2a105d7e64: Request end of tablet copy session 0d77752181e944698e809e325e5937e6-752e0d08505b4c8cb4b2a44cc00ce858 received from {username='slave'} at 127.0.99.136:34769
I20240503 14:00:00.582648 6811 tablet_copy_service.cc:433] P 4a85e492c8154e9881a9cc2a105d7e64: ending tablet copy session 0d77752181e944698e809e325e5937e6-752e0d08505b4c8cb4b2a44cc00ce858 on tablet 752e0d08505b4c8cb4b2a44cc00ce858 with peer 0d77752181e944698e809e325e5937e6
I20240503 14:00:00.585788 16232 tablet_bootstrap.cc:492] T 3856ff74b3bf4f608e3c6cdd06ce144e P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:00.586038 16232 tablet_bootstrap.cc:492] T 3856ff74b3bf4f608e3c6cdd06ce144e P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap complete.
I20240503 14:00:00.586087 16342 tablet_copy_client.cc:523] T 873f3615984646159e1b05178c933e18 P 2145d15ab2d74c0287839266f1411a43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:00.586220 16232 ts_tablet_manager.cc:1367] T 3856ff74b3bf4f608e3c6cdd06ce144e P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.064s user 0.018s sys 0.007s
I20240503 14:00:00.586843 16232 raft_consensus.cc:348] T 3856ff74b3bf4f608e3c6cdd06ce144e P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 14:00:00.586962 16232 raft_consensus.cc:729] T 3856ff74b3bf4f608e3c6cdd06ce144e P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: LEARNER
I20240503 14:00:00.587069 16232 consensus_queue.cc:260] T 3856ff74b3bf4f608e3c6cdd06ce144e P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 14:00:00.587363 16232 ts_tablet_manager.cc:1398] T 3856ff74b3bf4f608e3c6cdd06ce144e P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 14:00:00.587623 16342 tablet_bootstrap.cc:492] T 873f3615984646159e1b05178c933e18 P 2145d15ab2d74c0287839266f1411a43: Bootstrap starting.
I20240503 14:00:00.587780 6942 tablet_copy_service.cc:342] P f58065db70b64ef8b1dc499aa6ae4460: Request end of tablet copy session 06ca8dbc004c42b99faa527a4e6f9545-3856ff74b3bf4f608e3c6cdd06ce144e received from {username='slave'} at 127.0.99.133:60451
I20240503 14:00:00.587852 6942 tablet_copy_service.cc:433] P f58065db70b64ef8b1dc499aa6ae4460: ending tablet copy session 06ca8dbc004c42b99faa527a4e6f9545-3856ff74b3bf4f608e3c6cdd06ce144e on tablet 3856ff74b3bf4f608e3c6cdd06ce144e with peer 06ca8dbc004c42b99faa527a4e6f9545
I20240503 14:00:00.594677 16342 tablet_bootstrap.cc:492] T 873f3615984646159e1b05178c933e18 P 2145d15ab2d74c0287839266f1411a43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:00.594935 16342 tablet_bootstrap.cc:492] T 873f3615984646159e1b05178c933e18 P 2145d15ab2d74c0287839266f1411a43: Bootstrap complete.
I20240503 14:00:00.595432 16342 ts_tablet_manager.cc:1367] T 873f3615984646159e1b05178c933e18 P 2145d15ab2d74c0287839266f1411a43: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20240503 14:00:00.596077 16342 raft_consensus.cc:348] T 873f3615984646159e1b05178c933e18 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 14:00:00.596203 16342 raft_consensus.cc:729] T 873f3615984646159e1b05178c933e18 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Initialized, Role: LEARNER
I20240503 14:00:00.596302 16342 consensus_queue.cc:260] T 873f3615984646159e1b05178c933e18 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: true } }
I20240503 14:00:00.596635 16342 ts_tablet_manager.cc:1398] T 873f3615984646159e1b05178c933e18 P 2145d15ab2d74c0287839266f1411a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:00.597142 6942 tablet_copy_service.cc:342] P f58065db70b64ef8b1dc499aa6ae4460: Request end of tablet copy session 2145d15ab2d74c0287839266f1411a43-873f3615984646159e1b05178c933e18 received from {username='slave'} at 127.0.99.130:57151
I20240503 14:00:00.597224 6942 tablet_copy_service.cc:433] P f58065db70b64ef8b1dc499aa6ae4460: ending tablet copy session 2145d15ab2d74c0287839266f1411a43-873f3615984646159e1b05178c933e18 on tablet 873f3615984646159e1b05178c933e18 with peer 2145d15ab2d74c0287839266f1411a43
I20240503 14:00:00.631598 15243 raft_consensus.cc:1050] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da: attempting to promote NON_VOTER 0d77752181e944698e809e325e5937e6 to VOTER
I20240503 14:00:00.632072 15243 consensus_queue.cc:237] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 14:00:00.633610 8947 raft_consensus.cc:1261] T a9e87083385d4e6bac9bda95670b68a8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:00:00.633973 9205 raft_consensus.cc:1261] T a9e87083385d4e6bac9bda95670b68a8 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:00:00.634071 15243 consensus_queue.cc:1035] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:00:00.634308 9343 raft_consensus.cc:1261] T a9e87083385d4e6bac9bda95670b68a8 P 0d77752181e944698e809e325e5937e6 [term 1 LEARNER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:00:00.634375 15099 consensus_queue.cc:1035] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:00:00.634727 15099 consensus_queue.cc:1035] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:00:00.639631 15243 raft_consensus.cc:2932] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 14:00:00.640092 9205 raft_consensus.cc:2932] T a9e87083385d4e6bac9bda95670b68a8 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 14:00:00.641556 8946 raft_consensus.cc:2932] T a9e87083385d4e6bac9bda95670b68a8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 14:00:00.641846 9343 raft_consensus.cc:2932] T a9e87083385d4e6bac9bda95670b68a8 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 14:00:00.646397 6385 catalog_manager.cc:5484] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index 2 to 3, 0d77752181e944698e809e325e5937e6 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:00.649718 6527 consensus_queue.cc:237] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 14:00:00.650838 9342 raft_consensus.cc:1261] T a9e87083385d4e6bac9bda95670b68a8 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:00.651422 15416 consensus_queue.cc:1035] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:00.652585 8946 raft_consensus.cc:1261] T a9e87083385d4e6bac9bda95670b68a8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:00.653017 15972 consensus_queue.cc:1035] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:00.654806 16310 ts_tablet_manager.cc:902] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 0d77752181e944698e809e325e5937e6: Initiating tablet copy from peer 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
F20240503 14:00:00.654987 16310 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-00434c9ea1654f7a9a9c31397a71998d.queue_time_us, tablet-download-pool-00434c9ea1654f7a9a9c31397a71998d.queue_time_us) (tablet-download-pool-00434c9ea1654f7a9a9c31397a71998d.run_cpu_time_us, tablet-download-pool-00434c9ea1654f7a9a9c31397a71998d.run_cpu_time_us) (tablet-download-pool-00434c9ea1654f7a9a9c31397a71998d.run_wall_time_us, tablet-download-pool-00434c9ea1654f7a9a9c31397a71998d.run_wall_time_us) (tablet-download-pool-1ab63490264c4755aeab897f865e9a11.queue_time_us, tablet-download-pool-1ab63490264c4755aeab897f865e9a11.queue_time_us) (tablet-download-pool-1ab63490264c4755aeab897f865e9a11.run_cpu_time_us, tablet-download-pool-1ab63490264c4755aeab897f865e9a11.run_cpu_time_us) (tablet-download-pool-1ab63490264c4755aeab897f865e9a11.run_wall_time_us, tablet-download-pool-1ab63490264c4755aeab897f865e9a11.run_wall_time_us) (tablet-download-pool-342229593b184019ab915c0389b184b4.queue_time_us, tablet-download-pool-342229593b184019ab915c0389b184b4.queue_time_us) (tablet-download-pool-342229593b184019ab915c0389b184b4.run_cpu_time_us, tablet-download-pool-342229593b184019ab915c0389b184b4.run_cpu_time_us) (tablet-download-pool-342229593b184019ab915c0389b184b4.run_wall_time_us, tablet-download-pool-342229593b184019ab915c0389b184b4.run_wall_time_us) (tablet-download-pool-3612cd35d8fa4300a1132fe43a63e6c1.queue_time_us, tablet-download-pool-3612cd35d8fa4300a1132fe43a63e6c1.queue_time_us) (tablet-download-pool-3612cd35d8fa4300a1132fe43a63e6c1.run_cpu_time_us, tablet-download-pool-3612cd35d8fa4300a1132fe43a63e6c1.run_cpu_time_us) (tablet-download-pool-3612cd35d8fa4300a1132fe43a63e6c1.run_wall_time_us, tablet-download-pool-3612cd35d8fa4300a1132fe43a63e6c1.run_wall_time_us) (tablet-download-pool-752e0d08505b4c8cb4b2a44cc00ce858.queue_time_us, tablet-download-pool-752e0d08505b4c8cb4b2a44cc00ce858.queue_time_us) (tablet-download-pool-752e0d08505b4c8cb4b2a44cc00ce858.run_cpu_time_us, tablet-download-pool-752e0d08505b4c8cb4b2a44cc00ce858.run_cpu_time_us) (tablet-download-pool-752e0d08505b4c8cb4b2a44cc00ce858.run_wall_time_us, tablet-download-pool-752e0d08505b4c8cb4b2a44cc00ce858.run_wall_time_us) (tablet-download-pool-9a6709e4024d47448c20d33a2b36479e.queue_time_us, tablet-download-pool-9a6709e4024d47448c20d33a2b36479e.queue_time_us) (tablet-download-pool-9a6709e4024d47448c20d33a2b36479e.run_cpu_time_us, tablet-download-pool-9a6709e4024d47448c20d33a2b36479e.run_cpu_time_us) (tablet-download-pool-9a6709e4024d47448c20d33a2b36479e.run_wall_time_us, tablet-download-pool-9a6709e4024d47448c20d33a2b36479e.run_wall_time_us) (tablet-download-pool-a9e87083385d4e6bac9bda95670b68a8.queue_time_us, tablet-download-pool-a9e87083385d4e6bac9bda95670b68a8.queue_time_us) (tablet-download-pool-a9e87083385d4e6bac9bda95670b68a8.run_cpu_time_us, tablet-download-pool-a9e87083385d4e6bac9bda95670b68a8.run_cpu_time_us) (tablet-download-pool-a9e87083385d4e6bac9bda95670b68a8.run_wall_time_us, tablet-download-pool-a9e87083385d4e6bac9bda95670b68a8.run_wall_time_us) (tablet-download-pool-aaa93a7fd5df4cb7967d6bdb0c8dd6d1.queue_time_us, tablet-download-pool-aaa93a7fd5df4cb7967d6bdb0c8dd6d1.queue_time_us) (tablet-download-pool-af98fddcb623403c841a77ecc10a3e07.queue_time_us, tablet-download-pool-af98fddcb623403c841a77ecc10a3e07.queue_time_us) (tablet-download-pool-af98fddcb623403c841a77ecc10a3e07.run_cpu_time_us, tablet-download-pool-af98fddcb623403c841a77ecc10a3e07.run_cpu_time_us) (tablet-download-pool-af98fddcb623403c841a77ecc10a3e07.run_wall_time_us, tablet-download-pool-af98fddcb623403c841a77ecc10a3e07.run_wall_time_us) (tablet-download-pool-d55d270456d3460586ff3cd41787b5d2.queue_time_us, tablet-download-pool-d55d270456d3460586ff3cd41787b5d2.queue_time_us) (tablet-download-pool-d55d270456d3460586ff3cd41787b5d2.run_cpu_time_us, tablet-download-pool-d55d270456d3460586ff3cd41787b5d2.run_cpu_time_us) (tablet-download-pool-d55d270456d3460586ff3cd41787b5d2.run_wall_time_us, tablet-download-pool-d55d270456d3460586ff3cd41787b5d2.run_wall_time_us) (tablet-download-pool-dfa48440020d4754be0f5ddf159c23fe.queue_time_us, tablet-download-pool-dfa48440020d4754be0f5ddf159c23fe.queue_time_us) (tablet-download-pool-dfa48440020d4754be0f5ddf159c23fe.run_cpu_time_us, tablet-download-pool-dfa48440020d4754be0f5ddf159c23fe.run_cpu_time_us) (tablet-download-pool-dfa48440020d4754be0f5ddf159c23fe.run_wall_time_us, tablet-download-pool-dfa48440020d4754be0f5ddf159c23fe.run_wall_time_us) (tablet-download-pool-e106b1432baf423493c95fcce63a55da.queue_time_us, tablet-download-pool-e106b1432baf423493c95fcce63a55da.queue_time_us) (tablet-download-pool-e106b1432baf423493c95fcce63a55da.run_cpu_time_us, tablet-download-pool-e106b1432baf423493c95fcce63a55da.run_cpu_time_us) (tablet-download-pool-e106b1432baf423493c95fcce63a55da.run_wall_time_us, tablet-download-pool-e106b1432baf423493c95fcce63a55da.run_wall_time_us) (tablet-download-pool-ea45c42d7fed4171a1b726b0684d2fa5.queue_time_us, tablet-download-pool-ea45c42d7fed4171a1b726b0684d2fa5.queue_time_us) (tablet-download-pool-ea45c42d7fed4171a1b726b0684d2fa5.run_cpu_time_us, tablet-download-pool-ea45c42d7fed4171a1b726b0684d2fa5.run_cpu_time_us) (tablet-download-pool-ea45c42d7fed4171a1b726b0684d2fa5.run_wall_time_us, tablet-download-pool-ea45c42d7fed4171a1b726b0684d2fa5.run_wall_time_us) (tablet-download-pool-ef19b1e5f15e49448c1b2919a24090d3.queue_time_us, tablet-download-pool-ef19b1e5f15e49448c1b2919a24090d3.queue_time_us) (tablet-download-pool-ef19b1e5f15e49448c1b2919a24090d3.run_cpu_time_us, tablet-download-pool-ef19b1e5f15e49448c1b2919a24090d3.run_cpu_time_us) (tablet-download-pool-ef19b1e5f15e49448c1b2919a24090d3.run_wall_time_us, tablet-download-pool-ef19b1e5f15e49448c1b2919a24090d3.run_wall_time_us) (tablet-download-pool-f403dc06195d449e8f454a9f4e795925.queue_time_us, tablet-download-pool-f403dc06195d449e8f454a9f4e795925.queue_time_us) (tablet-download-pool-f403dc06195d449e8f454a9f4e795925.run_cpu_time_us, tablet-download-pool-f403dc06195d449e8f454a9f4e795925.run_cpu_time_us) (tablet-download-pool-f403dc06195d449e8f454a9f4e795925.run_wall_time_us, tablet-download-pool-f403dc06195d449e8f454a9f4e795925.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240503 14:00:00.655539 15972 raft_consensus.cc:2932] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 14:00:00.657176 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a9e87083385d4e6bac9bda95670b68a8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:00.657305 8946 raft_consensus.cc:2932] T a9e87083385d4e6bac9bda95670b68a8 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
@ 0x7fae095f0dcd google::LogMessage::Fail() at ??:0
I20240503 14:00:00.659003 9343 raft_consensus.cc:2932] T a9e87083385d4e6bac9bda95670b68a8 P 0d77752181e944698e809e325e5937e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
I20240503 14:00:00.661258 6387 catalog_manager.cc:5484] T a9e87083385d4e6bac9bda95670b68a8 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index 3 to 4, VOTER ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } } }
@ 0x7fae095f4b93 google::LogMessage::SendToLog() at ??:0
I20240503 14:00:00.666188 9192 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e87083385d4e6bac9bda95670b68a8 with delete_type TABLET_DATA_TOMBSTONED (TS ab1d21935d6049888eb0ceccc3c9d735 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57960
I20240503 14:00:00.666637 16407 tablet_replica.cc:330] T a9e87083385d4e6bac9bda95670b68a8 P ab1d21935d6049888eb0ceccc3c9d735: stopping tablet replica
I20240503 14:00:00.666788 16407 raft_consensus.cc:2229] T a9e87083385d4e6bac9bda95670b68a8 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:00.667732 16407 raft_consensus.cc:2258] T a9e87083385d4e6bac9bda95670b68a8 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:00.668417 16407 ts_tablet_manager.cc:1875] T a9e87083385d4e6bac9bda95670b68a8 P ab1d21935d6049888eb0ceccc3c9d735: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
@ 0x7fae095f07cc google::LogMessage::Flush() at ??:0
@ 0x7fae095f1f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240503 14:00:00.671773 16407 ts_tablet_manager.cc:1888] T a9e87083385d4e6bac9bda95670b68a8 P ab1d21935d6049888eb0ceccc3c9d735: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:00:00.671849 16407 log.cc:1196] T a9e87083385d4e6bac9bda95670b68a8 P ab1d21935d6049888eb0ceccc3c9d735: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/a9e87083385d4e6bac9bda95670b68a8
I20240503 14:00:00.672444 6373 catalog_manager.cc:4830] TS ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091): tablet a9e87083385d4e6bac9bda95670b68a8 (table rebalancer_extra_table_0 [id=0be76966adde4247b7969f0f470308fd]) successfully deleted
@ 0x7fae0aa5a044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fae0aa3ef98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fae0aa3cefc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fae0d5ef77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fae0d5f8a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fae0d67976f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fae0d68d4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fae0d678226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7fae0d687c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55e5e9ab775c std::function<>::operator()() at ??:0
@ 0x7fae0aa41e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fae0aa426ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fae0aa4403b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55e5e9ab775c std::function<>::operator()() at ??:0
@ 0x7fae0aa31807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fae0b26f6db start_thread at ??:0
@ 0x7fae087a771f clone at ??:0
I20240503 14:00:00.753759 9205 raft_consensus.cc:1203] T d51634c08e2749d4a3bbff4b2a786e48 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:00:00.772584 16442 ts_tablet_manager.cc:902] T 9af92a2d926543c4aa68a1f85b7da24b P 4a85e492c8154e9881a9cc2a105d7e64: Initiating tablet copy from peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 14:00:00.774163 16442 tablet_copy_client.cc:308] T 9af92a2d926543c4aa68a1f85b7da24b P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:34547
I20240503 14:00:00.774605 8968 tablet_copy_service.cc:140] P 06ca8dbc004c42b99faa527a4e6f9545: Received BeginTabletCopySession request for tablet 9af92a2d926543c4aa68a1f85b7da24b from peer 4a85e492c8154e9881a9cc2a105d7e64 ({username='slave'} at 127.0.99.131:54275)
I20240503 14:00:00.774710 8968 tablet_copy_service.cc:161] P 06ca8dbc004c42b99faa527a4e6f9545: Beginning new tablet copy session on tablet 9af92a2d926543c4aa68a1f85b7da24b from peer 4a85e492c8154e9881a9cc2a105d7e64 at {username='slave'} at 127.0.99.131:54275: session id = 4a85e492c8154e9881a9cc2a105d7e64-9af92a2d926543c4aa68a1f85b7da24b
I20240503 14:00:00.775614 8968 tablet_copy_source_session.cc:205] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:00.776800 16442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9af92a2d926543c4aa68a1f85b7da24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:00.779220 16442 tablet_copy_client.cc:783] T 9af92a2d926543c4aa68a1f85b7da24b P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:00.779394 16442 tablet_copy_client.cc:647] T 9af92a2d926543c4aa68a1f85b7da24b P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:00.781013 16442 tablet_copy_client.cc:523] T 9af92a2d926543c4aa68a1f85b7da24b P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:00.782392 16442 tablet_bootstrap.cc:492] T 9af92a2d926543c4aa68a1f85b7da24b P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
I20240503 14:00:00.785876 16442 tablet_bootstrap.cc:492] T 9af92a2d926543c4aa68a1f85b7da24b P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:00.786118 16442 tablet_bootstrap.cc:492] T 9af92a2d926543c4aa68a1f85b7da24b P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap complete.
I20240503 14:00:00.786265 16442 ts_tablet_manager.cc:1367] T 9af92a2d926543c4aa68a1f85b7da24b P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 14:00:00.786687 16442 raft_consensus.cc:348] T 9af92a2d926543c4aa68a1f85b7da24b P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 14:00:00.786803 16442 raft_consensus.cc:729] T 9af92a2d926543c4aa68a1f85b7da24b P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: LEARNER
I20240503 14:00:00.786897 16442 consensus_queue.cc:260] T 9af92a2d926543c4aa68a1f85b7da24b P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 14:00:00.788252 16442 ts_tablet_manager.cc:1398] T 9af92a2d926543c4aa68a1f85b7da24b P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:00:00.788674 8968 tablet_copy_service.cc:342] P 06ca8dbc004c42b99faa527a4e6f9545: Request end of tablet copy session 4a85e492c8154e9881a9cc2a105d7e64-9af92a2d926543c4aa68a1f85b7da24b received from {username='slave'} at 127.0.99.131:54275
I20240503 14:00:00.788755 8968 tablet_copy_service.cc:433] P 06ca8dbc004c42b99faa527a4e6f9545: ending tablet copy session 4a85e492c8154e9881a9cc2a105d7e64-9af92a2d926543c4aa68a1f85b7da24b on tablet 9af92a2d926543c4aa68a1f85b7da24b with peer 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:00.807762 8946 raft_consensus.cc:1203] T b6cb5119a6db403ba3cae9fa9b121211 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240503 14:00:00.948235 6858 connection.cc:535] client connection to 127.0.99.136:41931 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:00:00.948246 6596 connection.cc:535] client connection to 127.0.99.136:41931 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:00:00.948344 9017 connection.cc:535] client connection to 127.0.99.136:41931 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:00:00.948416 6596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:00:00.948416 6858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:00:00.948437 9017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:00:00.949118 6465 connection.cc:535] client connection to 127.0.99.136:41931 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:00:00.949239 6465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:00:00.949635 8884 connection.cc:535] client connection to 127.0.99.136:41931 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:00:00.949733 8884 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:00:00.953666 6858 consensus_peers.cc:485] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:00.953737 6858 consensus_peers.cc:485] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:00.954223 9017 consensus_peers.cc:485] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:00.954290 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:00.954322 9017 consensus_peers.cc:485] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:00.954355 9017 consensus_peers.cc:485] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:00.954388 9017 consensus_peers.cc:485] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:00.957005 6465 consensus_peers.cc:485] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:00.957800 6727 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:00.958647 6465 consensus_peers.cc:485] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:00.958726 6465 consensus_peers.cc:485] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:00.959028 6727 consensus_peers.cc:485] T d55d270456d3460586ff3cd41787b5d2 P 4a85e492c8154e9881a9cc2a105d7e64 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:00.959070 6596 consensus_peers.cc:485] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:00.959158 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:00.959733 8884 consensus_peers.cc:485] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:00.959813 8884 consensus_peers.cc:485] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:00.962210 9148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:00.962400 9205 consensus_queue.cc:237] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
W20240503 14:00:00.963541 9148 consensus_peers.cc:485] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:00.964514 9081 raft_consensus.cc:1261] T 4825989d826642bbb5f4bf0ea350e27c P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:00.964875 6526 raft_consensus.cc:1261] T 4825989d826642bbb5f4bf0ea350e27c P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:00.964883 16096 consensus_queue.cc:1035] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:00:00.965193 16096 consensus_queue.cc:1035] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:00:00.965356 9146 consensus_peers.cc:485] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4825989d826642bbb5f4bf0ea350e27c. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:00.966740 12063 raft_consensus.cc:2932] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 14:00:00.967382 9081 raft_consensus.cc:2932] T 4825989d826642bbb5f4bf0ea350e27c P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 14:00:00.969929 6526 raft_consensus.cc:2932] T 4825989d826642bbb5f4bf0ea350e27c P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } } }
I20240503 14:00:00.972435 6387 catalog_manager.cc:5484] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: config changed from index -1 to 2, NON_VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:00.984828 8946 raft_consensus.cc:1203] T 3856ff74b3bf4f608e3c6cdd06ce144e P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:00.984995 6789 consensus_queue.cc:237] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 14:00:00.987835 6920 raft_consensus.cc:1261] T 95fb9ad43cf34a78b98965035cd086ec P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:00.988261 6658 raft_consensus.cc:1261] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer 4a85e492c8154e9881a9cc2a105d7e64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:00.988297 14879 consensus_queue.cc:1035] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:00.988778 14879 consensus_queue.cc:1035] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:00:00.989378 6726 consensus_peers.cc:485] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64 -> Peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): Couldn't send request to peer 06ca8dbc004c42b99faa527a4e6f9545. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 95fb9ad43cf34a78b98965035cd086ec. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:00.990980 15149 raft_consensus.cc:2932] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 14:00:00.991464 6658 raft_consensus.cc:2932] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
I20240503 14:00:00.994864 6379 catalog_manager.cc:5484] T 95fb9ad43cf34a78b98965035cd086ec P 4a85e492c8154e9881a9cc2a105d7e64 reported cstate change: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "4a85e492c8154e9881a9cc2a105d7e64" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:00.996448 6920 raft_consensus.cc:2932] T 95fb9ad43cf34a78b98965035cd086ec P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } } }
W20240503 14:00:01.001039 9148 consensus_peers.cc:485] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.002306 9017 consensus_peers.cc:485] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:01.003316 6658 raft_consensus.cc:1203] T 873f3615984646159e1b05178c933e18 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240503 14:00:01.016170 6858 consensus_peers.cc:485] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.023597 9017 consensus_peers.cc:485] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.043695 8884 consensus_peers.cc:485] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.048930 9017 consensus_peers.cc:485] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.069907 6727 consensus_peers.cc:485] T 6ba45113776347c3adeae091c95427df P 4a85e492c8154e9881a9cc2a105d7e64 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.076996 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.081496 16232 ts_tablet_manager.cc:902] T 586369c0c3174070a3e184af38b3adf7 P 06ca8dbc004c42b99faa527a4e6f9545: Initiating tablet copy from peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 14:00:01.082137 16232 tablet_copy_client.cc:308] T 586369c0c3174070a3e184af38b3adf7 P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:46365
I20240503 14:00:01.082749 6811 tablet_copy_service.cc:140] P 4a85e492c8154e9881a9cc2a105d7e64: Received BeginTabletCopySession request for tablet 586369c0c3174070a3e184af38b3adf7 from peer 06ca8dbc004c42b99faa527a4e6f9545 ({username='slave'} at 127.0.99.133:36601)
I20240503 14:00:01.082888 6811 tablet_copy_service.cc:161] P 4a85e492c8154e9881a9cc2a105d7e64: Beginning new tablet copy session on tablet 586369c0c3174070a3e184af38b3adf7 from peer 06ca8dbc004c42b99faa527a4e6f9545 at {username='slave'} at 127.0.99.133:36601: session id = 06ca8dbc004c42b99faa527a4e6f9545-586369c0c3174070a3e184af38b3adf7
I20240503 14:00:01.084231 6811 tablet_copy_source_session.cc:205] T 586369c0c3174070a3e184af38b3adf7 P 4a85e492c8154e9881a9cc2a105d7e64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:01.085209 16232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 586369c0c3174070a3e184af38b3adf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:01.087364 16232 tablet_copy_client.cc:783] T 586369c0c3174070a3e184af38b3adf7 P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:01.087532 16232 tablet_copy_client.cc:647] T 586369c0c3174070a3e184af38b3adf7 P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:01.088797 16232 tablet_copy_client.cc:523] T 586369c0c3174070a3e184af38b3adf7 P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:01.089910 16232 tablet_bootstrap.cc:492] T 586369c0c3174070a3e184af38b3adf7 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 14:00:01.094419 16442 ts_tablet_manager.cc:902] T 564582aa33a7465081724523096591d3 P 4a85e492c8154e9881a9cc2a105d7e64: Initiating tablet copy from peer 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 14:00:01.094898 16442 tablet_copy_client.cc:308] T 564582aa33a7465081724523096591d3 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46119
I20240503 14:00:01.095355 6680 tablet_copy_service.cc:140] P 2145d15ab2d74c0287839266f1411a43: Received BeginTabletCopySession request for tablet 564582aa33a7465081724523096591d3 from peer 4a85e492c8154e9881a9cc2a105d7e64 ({username='slave'} at 127.0.99.131:59523)
I20240503 14:00:01.095450 6680 tablet_copy_service.cc:161] P 2145d15ab2d74c0287839266f1411a43: Beginning new tablet copy session on tablet 564582aa33a7465081724523096591d3 from peer 4a85e492c8154e9881a9cc2a105d7e64 at {username='slave'} at 127.0.99.131:59523: session id = 4a85e492c8154e9881a9cc2a105d7e64-564582aa33a7465081724523096591d3
I20240503 14:00:01.096521 6680 tablet_copy_source_session.cc:205] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:01.097469 16442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 564582aa33a7465081724523096591d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:01.099486 16442 tablet_copy_client.cc:783] T 564582aa33a7465081724523096591d3 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:01.099678 16442 tablet_copy_client.cc:647] T 564582aa33a7465081724523096591d3 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:01.102545 16442 tablet_copy_client.cc:523] T 564582aa33a7465081724523096591d3 P 4a85e492c8154e9881a9cc2a105d7e64: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:01.104115 16442 tablet_bootstrap.cc:492] T 564582aa33a7465081724523096591d3 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap starting.
W20240503 14:00:01.105235 6596 consensus_peers.cc:485] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:01.109349 16442 tablet_bootstrap.cc:492] T 564582aa33a7465081724523096591d3 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:01.109591 16442 tablet_bootstrap.cc:492] T 564582aa33a7465081724523096591d3 P 4a85e492c8154e9881a9cc2a105d7e64: Bootstrap complete.
I20240503 14:00:01.109755 16442 ts_tablet_manager.cc:1367] T 564582aa33a7465081724523096591d3 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 14:00:01.110214 16442 raft_consensus.cc:348] T 564582aa33a7465081724523096591d3 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 14:00:01.110352 16442 raft_consensus.cc:729] T 564582aa33a7465081724523096591d3 P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4a85e492c8154e9881a9cc2a105d7e64, State: Initialized, Role: LEARNER
I20240503 14:00:01.110461 16442 consensus_queue.cc:260] T 564582aa33a7465081724523096591d3 P 4a85e492c8154e9881a9cc2a105d7e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: true } }
I20240503 14:00:01.110766 16442 ts_tablet_manager.cc:1398] T 564582aa33a7465081724523096591d3 P 4a85e492c8154e9881a9cc2a105d7e64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:01.112543 6680 tablet_copy_service.cc:342] P 2145d15ab2d74c0287839266f1411a43: Request end of tablet copy session 4a85e492c8154e9881a9cc2a105d7e64-564582aa33a7465081724523096591d3 received from {username='slave'} at 127.0.99.131:59523
I20240503 14:00:01.112622 6680 tablet_copy_service.cc:433] P 2145d15ab2d74c0287839266f1411a43: ending tablet copy session 4a85e492c8154e9881a9cc2a105d7e64-564582aa33a7465081724523096591d3 on tablet 564582aa33a7465081724523096591d3 with peer 4a85e492c8154e9881a9cc2a105d7e64
W20240503 14:00:01.113745 6858 consensus_peers.cc:485] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:01.113837 16232 tablet_bootstrap.cc:492] T 586369c0c3174070a3e184af38b3adf7 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:01.114147 16232 tablet_bootstrap.cc:492] T 586369c0c3174070a3e184af38b3adf7 P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap complete.
I20240503 14:00:01.114408 16232 ts_tablet_manager.cc:1367] T 586369c0c3174070a3e184af38b3adf7 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.025s user 0.024s sys 0.000s
I20240503 14:00:01.114863 16232 raft_consensus.cc:348] T 586369c0c3174070a3e184af38b3adf7 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 14:00:01.114986 16232 raft_consensus.cc:729] T 586369c0c3174070a3e184af38b3adf7 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: LEARNER
I20240503 14:00:01.115072 16232 consensus_queue.cc:260] T 586369c0c3174070a3e184af38b3adf7 P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 14:00:01.115445 16232 ts_tablet_manager.cc:1398] T 586369c0c3174070a3e184af38b3adf7 P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:01.116196 6811 tablet_copy_service.cc:342] P 4a85e492c8154e9881a9cc2a105d7e64: Request end of tablet copy session 06ca8dbc004c42b99faa527a4e6f9545-586369c0c3174070a3e184af38b3adf7 received from {username='slave'} at 127.0.99.133:36601
I20240503 14:00:01.116273 6811 tablet_copy_service.cc:433] P 4a85e492c8154e9881a9cc2a105d7e64: ending tablet copy session 06ca8dbc004c42b99faa527a4e6f9545-586369c0c3174070a3e184af38b3adf7 on tablet 586369c0c3174070a3e184af38b3adf7 with peer 06ca8dbc004c42b99faa527a4e6f9545
W20240503 14:00:01.142433 6858 consensus_peers.cc:485] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.143121 9148 consensus_peers.cc:485] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.152190 6727 consensus_peers.cc:485] T ea45c42d7fed4171a1b726b0684d2fa5 P 4a85e492c8154e9881a9cc2a105d7e64 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.161698 9148 consensus_peers.cc:485] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:01.167619 14384 raft_consensus.cc:1050] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545: attempting to promote NON_VOTER ab1d21935d6049888eb0ceccc3c9d735 to VOTER
I20240503 14:00:01.168020 14384 consensus_queue.cc:237] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 14:00:01.169343 9205 raft_consensus.cc:1261] T d51634c08e2749d4a3bbff4b2a786e48 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEARNER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:00:01.169350 9081 raft_consensus.cc:1261] T d51634c08e2749d4a3bbff4b2a786e48 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:00:01.169350 6920 raft_consensus.cc:1261] T d51634c08e2749d4a3bbff4b2a786e48 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:00:01.169697 12367 consensus_queue.cc:1035] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:00:01.169867 14390 consensus_queue.cc:1035] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:00:01.170034 16564 consensus_queue.cc:1035] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:00:01.171805 12367 raft_consensus.cc:2932] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 14:00:01.172335 9205 raft_consensus.cc:2932] T d51634c08e2749d4a3bbff4b2a786e48 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 14:00:01.172355 6920 raft_consensus.cc:2932] T d51634c08e2749d4a3bbff4b2a786e48 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 14:00:01.174592 9081 raft_consensus.cc:2932] T d51634c08e2749d4a3bbff4b2a786e48 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 14:00:01.175768 6387 catalog_manager.cc:5484] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index 2 to 3, ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:01.178915 8946 consensus_queue.cc:237] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 14:00:01.180260 9205 raft_consensus.cc:1261] T d51634c08e2749d4a3bbff4b2a786e48 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:01.180943 16564 consensus_queue.cc:1035] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:01.182377 14384 raft_consensus.cc:2932] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 14:00:01.182824 6920 raft_consensus.cc:1261] T d51634c08e2749d4a3bbff4b2a786e48 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:01.183650 9205 raft_consensus.cc:2932] T d51634c08e2749d4a3bbff4b2a786e48 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 14:00:01.183853 14384 consensus_queue.cc:1035] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:01.184221 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d51634c08e2749d4a3bbff4b2a786e48 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:01.185041 6920 raft_consensus.cc:2932] T d51634c08e2749d4a3bbff4b2a786e48 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } } }
I20240503 14:00:01.186720 6380 catalog_manager.cc:5484] T d51634c08e2749d4a3bbff4b2a786e48 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 14:00:01.189002 9017 consensus_peers.cc:485] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:01.189513 9060 tablet_service.cc:1514] Processing DeleteTablet for tablet d51634c08e2749d4a3bbff4b2a786e48 with delete_type TABLET_DATA_TOMBSTONED (TS 8d574fb4a89d4eea947c859470d44dd4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56018
I20240503 14:00:01.189798 16568 tablet_replica.cc:330] T d51634c08e2749d4a3bbff4b2a786e48 P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 14:00:01.189942 16568 raft_consensus.cc:2229] T d51634c08e2749d4a3bbff4b2a786e48 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:01.190090 16568 raft_consensus.cc:2258] T d51634c08e2749d4a3bbff4b2a786e48 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:01.190677 16568 ts_tablet_manager.cc:1875] T d51634c08e2749d4a3bbff4b2a786e48 P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:01.192550 16568 ts_tablet_manager.cc:1888] T d51634c08e2749d4a3bbff4b2a786e48 P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:00:01.192625 16568 log.cc:1196] T d51634c08e2749d4a3bbff4b2a786e48 P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/d51634c08e2749d4a3bbff4b2a786e48
I20240503 14:00:01.193114 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet d51634c08e2749d4a3bbff4b2a786e48 (table rebalancer_extra_table_16 [id=82c39415abdc4fe8a6c8d9f312c84cbc]) successfully deleted
I20240503 14:00:01.200800 14177 raft_consensus.cc:1050] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43: attempting to promote NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 to VOTER
I20240503 14:00:01.201278 14177 consensus_queue.cc:237] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
W20240503 14:00:01.201905 6465 consensus_peers.cc:485] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:01.202740 6526 raft_consensus.cc:1261] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:01.202888 8946 raft_consensus.cc:1261] T b6cb5119a6db403ba3cae9fa9b121211 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:01.203217 15959 consensus_queue.cc:1035] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:01.203593 15959 consensus_queue.cc:1035] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:01.204098 6920 raft_consensus.cc:1261] T b6cb5119a6db403ba3cae9fa9b121211 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:01.204546 15959 consensus_queue.cc:1035] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:01.205327 15876 raft_consensus.cc:2932] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 14:00:01.205947 8946 raft_consensus.cc:2932] T b6cb5119a6db403ba3cae9fa9b121211 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
W20240503 14:00:01.208355 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.209287 6385 catalog_manager.cc:5484] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:01.209522 6526 raft_consensus.cc:2932] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 14:00:01.210721 6920 raft_consensus.cc:2932] T b6cb5119a6db403ba3cae9fa9b121211 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
W20240503 14:00:01.219717 9017 consensus_peers.cc:485] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.222978 9148 consensus_peers.cc:485] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.236019 6596 consensus_peers.cc:485] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:01.252672 6789 raft_consensus.cc:1203] T 9af92a2d926543c4aa68a1f85b7da24b P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240503 14:00:01.314900 6465 consensus_peers.cc:485] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:01.389778 6659 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b6cb5119a6db403ba3cae9fa9b121211"
dest_uuid: "2145d15ab2d74c0287839266f1411a43"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56424
I20240503 14:00:01.389955 6659 raft_consensus.cc:595] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:01.412966 16442 ts_tablet_manager.cc:902] T 4825989d826642bbb5f4bf0ea350e27c P 4a85e492c8154e9881a9cc2a105d7e64: Initiating tablet copy from peer ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
F20240503 14:00:01.413200 16442 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-1350225b1a2848debca68e1bad5528c5.queue_time_us, tablet-download-pool-1350225b1a2848debca68e1bad5528c5.queue_time_us) (tablet-download-pool-1350225b1a2848debca68e1bad5528c5.run_cpu_time_us, tablet-download-pool-1350225b1a2848debca68e1bad5528c5.run_cpu_time_us) (tablet-download-pool-1350225b1a2848debca68e1bad5528c5.run_wall_time_us, tablet-download-pool-1350225b1a2848debca68e1bad5528c5.run_wall_time_us) (tablet-download-pool-4825989d826642bbb5f4bf0ea350e27c.queue_time_us, tablet-download-pool-4825989d826642bbb5f4bf0ea350e27c.queue_time_us) (tablet-download-pool-4a1398d983c54a7498257f766052e8e9.queue_time_us, tablet-download-pool-4a1398d983c54a7498257f766052e8e9.queue_time_us) (tablet-download-pool-4a1398d983c54a7498257f766052e8e9.run_cpu_time_us, tablet-download-pool-4a1398d983c54a7498257f766052e8e9.run_cpu_time_us) (tablet-download-pool-4a1398d983c54a7498257f766052e8e9.run_wall_time_us, tablet-download-pool-4a1398d983c54a7498257f766052e8e9.run_wall_time_us) (tablet-download-pool-564582aa33a7465081724523096591d3.queue_time_us, tablet-download-pool-564582aa33a7465081724523096591d3.queue_time_us) (tablet-download-pool-564582aa33a7465081724523096591d3.run_cpu_time_us, tablet-download-pool-564582aa33a7465081724523096591d3.run_cpu_time_us) (tablet-download-pool-564582aa33a7465081724523096591d3.run_wall_time_us, tablet-download-pool-564582aa33a7465081724523096591d3.run_wall_time_us) (tablet-download-pool-568c2e7206f9496baa22432b72c021b1.queue_time_us, tablet-download-pool-568c2e7206f9496baa22432b72c021b1.queue_time_us) (tablet-download-pool-568c2e7206f9496baa22432b72c021b1.run_cpu_time_us, tablet-download-pool-568c2e7206f9496baa22432b72c021b1.run_cpu_time_us) (tablet-download-pool-568c2e7206f9496baa22432b72c021b1.run_wall_time_us, tablet-download-pool-568c2e7206f9496baa22432b72c021b1.run_wall_time_us) (tablet-download-pool-62c77ec99cf443d2b7d48b12c8855883.queue_time_us, tablet-download-pool-62c77ec99cf443d2b7d48b12c8855883.queue_time_us) (tablet-download-pool-62c77ec99cf443d2b7d48b12c8855883.run_cpu_time_us, tablet-download-pool-62c77ec99cf443d2b7d48b12c8855883.run_cpu_time_us) (tablet-download-pool-62c77ec99cf443d2b7d48b12c8855883.run_wall_time_us, tablet-download-pool-62c77ec99cf443d2b7d48b12c8855883.run_wall_time_us) (tablet-download-pool-7a25af1e7ae64f4fad8ad82faa1c6eb6.queue_time_us, tablet-download-pool-7a25af1e7ae64f4fad8ad82faa1c6eb6.queue_time_us) (tablet-download-pool-7a25af1e7ae64f4fad8ad82faa1c6eb6.run_cpu_time_us, tablet-download-pool-7a25af1e7ae64f4fad8ad82faa1c6eb6.run_cpu_time_us) (tablet-download-pool-7a25af1e7ae64f4fad8ad82faa1c6eb6.run_wall_time_us, tablet-download-pool-7a25af1e7ae64f4fad8ad82faa1c6eb6.run_wall_time_us) (tablet-download-pool-7a44f5349cc44be69e9b52f8047d2bb1.queue_time_us, tablet-download-pool-7a44f5349cc44be69e9b52f8047d2bb1.queue_time_us) (tablet-download-pool-7a44f5349cc44be69e9b52f8047d2bb1.run_cpu_time_us, tablet-download-pool-7a44f5349cc44be69e9b52f8047d2bb1.run_cpu_time_us) (tablet-download-pool-7a44f5349cc44be69e9b52f8047d2bb1.run_wall_time_us, tablet-download-pool-7a44f5349cc44be69e9b52f8047d2bb1.run_wall_time_us) (tablet-download-pool-9af92a2d926543c4aa68a1f85b7da24b.queue_time_us, tablet-download-pool-9af92a2d926543c4aa68a1f85b7da24b.queue_time_us) (tablet-download-pool-9af92a2d926543c4aa68a1f85b7da24b.run_cpu_time_us, tablet-download-pool-9af92a2d926543c4aa68a1f85b7da24b.run_cpu_time_us) (tablet-download-pool-9af92a2d926543c4aa68a1f85b7da24b.run_wall_time_us, tablet-download-pool-9af92a2d926543c4aa68a1f85b7da24b.run_wall_time_us) (tablet-download-pool-9f0daf41253f4678b158e3ccbb7b0c98.queue_time_us, tablet-download-pool-9f0daf41253f4678b158e3ccbb7b0c98.queue_time_us) (tablet-download-pool-9f0daf41253f4678b158e3ccbb7b0c98.run_cpu_time_us, tablet-download-pool-9f0daf41253f4678b158e3ccbb7b0c98.run_cpu_time_us) (tablet-download-pool-9f0daf41253f4678b158e3ccbb7b0c98.run_wall_time_us, tablet-download-pool-9f0daf41253f4678b158e3ccbb7b0c98.run_wall_time_us) (tablet-download-pool-a7e9c80dbf8c41469e54535c20648df4.queue_time_us, tablet-download-pool-a7e9c80dbf8c41469e54535c20648df4.queue_time_us) (tablet-download-pool-a7e9c80dbf8c41469e54535c20648df4.run_cpu_time_us, tablet-download-pool-a7e9c80dbf8c41469e54535c20648df4.run_cpu_time_us) (tablet-download-pool-a7e9c80dbf8c41469e54535c20648df4.run_wall_time_us, tablet-download-pool-a7e9c80dbf8c41469e54535c20648df4.run_wall_time_us) (tablet-download-pool-ab6fe502a80c410a9b6ee6f5d3d3205e.queue_time_us, tablet-download-pool-ab6fe502a80c410a9b6ee6f5d3d3205e.queue_time_us) (tablet-download-pool-ab6fe502a80c410a9b6ee6f5d3d3205e.run_cpu_time_us, tablet-download-pool-ab6fe502a80c410a9b6ee6f5d3d3205e.run_cpu_time_us) (tablet-download-pool-ab6fe502a80c410a9b6ee6f5d3d3205e.run_wall_time_us, tablet-download-pool-ab6fe502a80c410a9b6ee6f5d3d3205e.run_wall_time_us) (tablet-download-pool-af8bba02684c4555893def9c1e88ebf0.queue_time_us, tablet-download-pool-af8bba02684c4555893def9c1e88ebf0.queue_time_us) (tablet-download-pool-af8bba02684c4555893def9c1e88ebf0.run_cpu_time_us, tablet-download-pool-af8bba02684c4555893def9c1e88ebf0.run_cpu_time_us) (tablet-download-pool-af8bba02684c4555893def9c1e88ebf0.run_wall_time_us, tablet-download-pool-af8bba02684c4555893def9c1e88ebf0.run_wall_time_us) (tablet-download-pool-d002f7a57be04915b7b0b3dcb985f550.queue_time_us, tablet-download-pool-d002f7a57be04915b7b0b3dcb985f550.queue_time_us) (tablet-download-pool-d002f7a57be04915b7b0b3dcb985f550.run_cpu_time_us, tablet-download-pool-d002f7a57be04915b7b0b3dcb985f550.run_cpu_time_us) (tablet-download-pool-d002f7a57be04915b7b0b3dcb985f550.run_wall_time_us, tablet-download-pool-d002f7a57be04915b7b0b3dcb985f550.run_wall_time_us) (tablet-download-pool-fd45a0cf3a2d4657b02016bc60c5be34.queue_time_us, tablet-download-pool-fd45a0cf3a2d4657b02016bc60c5be34.queue_time_us) (tablet-download-pool-fd45a0cf3a2d4657b02016bc60c5be34.run_cpu_time_us, tablet-download-pool-fd45a0cf3a2d4657b02016bc60c5be34.run_cpu_time_us) (tablet-download-pool-fd45a0cf3a2d4657b02016bc60c5be34.run_wall_time_us, tablet-download-pool-fd45a0cf3a2d4657b02016bc60c5be34.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f3bc692bdcd google::LogMessage::Fail() at ??:0
@ 0x7f3bc692fb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f3bc692b7cc google::LogMessage::Flush() at ??:0
I20240503 14:00:01.424705 15111 raft_consensus.cc:1050] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460: attempting to promote NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 to VOTER
I20240503 14:00:01.425132 15111 consensus_queue.cc:237] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
I20240503 14:00:01.426525 8946 raft_consensus.cc:1261] T 3856ff74b3bf4f608e3c6cdd06ce144e P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:01.426712 6659 raft_consensus.cc:1261] T 3856ff74b3bf4f608e3c6cdd06ce144e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
@ 0x7f3bc692cf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240503 14:00:01.426939 6788 raft_consensus.cc:1261] T 3856ff74b3bf4f608e3c6cdd06ce144e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:01.427100 12748 consensus_queue.cc:1035] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:01.427443 15927 consensus_queue.cc:1035] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:01.427695 13370 consensus_queue.cc:1035] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
@ 0x7f3bc7d95044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f3bc7d79f98 kudu::ThreadPool::ThreadPool() at ??:0
I20240503 14:00:01.438335 13370 raft_consensus.cc:2932] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
@ 0x7f3bc7d77efc kudu::ThreadPoolBuilder::Build() at ??:0
I20240503 14:00:01.441165 6788 raft_consensus.cc:2932] T 3856ff74b3bf4f608e3c6cdd06ce144e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 14:00:01.442021 8946 raft_consensus.cc:2932] T 3856ff74b3bf4f608e3c6cdd06ce144e P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 14:00:01.443068 6659 raft_consensus.cc:2932] T 3856ff74b3bf4f608e3c6cdd06ce144e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 14:00:01.446051 6380 catalog_manager.cc:5484] T 3856ff74b3bf4f608e3c6cdd06ce144e P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: config changed from index 4 to 5, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
@ 0x7f3bca92a77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f3bca933a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240503 14:00:01.452052 6922 consensus_queue.cc:237] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
I20240503 14:00:01.453794 8946 raft_consensus.cc:1261] T 3856ff74b3bf4f608e3c6cdd06ce144e P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:01.454577 15111 consensus_queue.cc:1035] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
@ 0x7f3bca9b476f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 14:00:01.458403 6659 raft_consensus.cc:1261] T 3856ff74b3bf4f608e3c6cdd06ce144e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:01.458868 12748 consensus_queue.cc:1035] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
@ 0x7f3bca9c84f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f3bca9b3226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240503 14:00:01.464176 12748 raft_consensus.cc:2932] T 3856ff74b3bf4f608e3c6cdd06ce144e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
@ 0x7f3bca9c2c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55bc92ba275c std::function<>::operator()() at ??:0
I20240503 14:00:01.469254 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3856ff74b3bf4f608e3c6cdd06ce144e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
@ 0x7f3bc7d7ce06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3bc7d7d6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240503 14:00:01.472100 6659 raft_consensus.cc:2932] T 3856ff74b3bf4f608e3c6cdd06ce144e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
@ 0x7f3bc7d7f03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240503 14:00:01.475036 8946 raft_consensus.cc:2932] T 3856ff74b3bf4f608e3c6cdd06ce144e P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
@ 0x55bc92ba275c std::function<>::operator()() at ??:0
I20240503 14:00:01.477129 6385 catalog_manager.cc:5484] T 3856ff74b3bf4f608e3c6cdd06ce144e P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index 5 to 6, VOTER 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
@ 0x7f3bc7d6c807 kudu::Thread::SuperviseThread() at ??:0
I20240503 14:00:01.481784 6769 tablet_service.cc:1514] Processing DeleteTablet for tablet 3856ff74b3bf4f608e3c6cdd06ce144e with delete_type TABLET_DATA_TOMBSTONED (TS 4a85e492c8154e9881a9cc2a105d7e64 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44858
@ 0x7f3bc85aa6db start_thread at ??:0
I20240503 14:00:01.484692 16683 tablet_replica.cc:330] T 3856ff74b3bf4f608e3c6cdd06ce144e P 4a85e492c8154e9881a9cc2a105d7e64: stopping tablet replica
I20240503 14:00:01.485186 16683 raft_consensus.cc:2229] T 3856ff74b3bf4f608e3c6cdd06ce144e P 4a85e492c8154e9881a9cc2a105d7e64 [term 1 FOLLOWER]: Raft consensus shutting down.
@ 0x7f3bc5ae271f clone at ??:0
W20240503 14:00:01.555548 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.617148 12748 raft_consensus.cc:1050] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460: attempting to promote NON_VOTER 2145d15ab2d74c0287839266f1411a43 to VOTER
I20240503 14:00:01.617560 12748 consensus_queue.cc:237] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 14:00:01.619052 8946 raft_consensus.cc:1261] T 873f3615984646159e1b05178c933e18 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:00:01.619184 6659 raft_consensus.cc:1261] T 873f3615984646159e1b05178c933e18 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEARNER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:00:01.619187 9081 raft_consensus.cc:1261] T 873f3615984646159e1b05178c933e18 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:00:01.619642 13370 consensus_queue.cc:1035] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:00:01.619822 15927 consensus_queue.cc:1035] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:00:01.620146 15111 consensus_queue.cc:1035] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:00:01.623608 15111 raft_consensus.cc:2932] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 14:00:01.625470 9081 raft_consensus.cc:2932] T 873f3615984646159e1b05178c933e18 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 14:00:01.626022 8946 raft_consensus.cc:2932] T 873f3615984646159e1b05178c933e18 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
W20240503 14:00:01.627353 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.628892 6385 catalog_manager.cc:5484] T 873f3615984646159e1b05178c933e18 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 14:00:01.641335 6659 raft_consensus.cc:2932] T 873f3615984646159e1b05178c933e18 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 14:00:01.646656 6922 consensus_queue.cc:237] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 14:00:01.648007 6659 raft_consensus.cc:1261] T 873f3615984646159e1b05178c933e18 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:01.648029 8946 raft_consensus.cc:1261] T 873f3615984646159e1b05178c933e18 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:01.648396 12748 consensus_queue.cc:1035] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:01.648622 15927 consensus_queue.cc:1035] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:01.650135 15927 raft_consensus.cc:2932] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 14:00:01.650983 8946 raft_consensus.cc:2932] T 873f3615984646159e1b05178c933e18 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 14:00:01.652585 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 873f3615984646159e1b05178c933e18 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:01.654187 6659 raft_consensus.cc:2932] T 873f3615984646159e1b05178c933e18 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } } }
I20240503 14:00:01.656112 6380 catalog_manager.cc:5484] T 873f3615984646159e1b05178c933e18 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: config changed from index 3 to 4, VOTER 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:01.659736 9060 tablet_service.cc:1514] Processing DeleteTablet for tablet 873f3615984646159e1b05178c933e18 with delete_type TABLET_DATA_TOMBSTONED (TS 8d574fb4a89d4eea947c859470d44dd4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56018
I20240503 14:00:01.659951 16568 tablet_replica.cc:330] T 873f3615984646159e1b05178c933e18 P 8d574fb4a89d4eea947c859470d44dd4: stopping tablet replica
I20240503 14:00:01.660197 16568 raft_consensus.cc:2229] T 873f3615984646159e1b05178c933e18 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:01.660393 16568 raft_consensus.cc:2258] T 873f3615984646159e1b05178c933e18 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:01.661227 16568 ts_tablet_manager.cc:1875] T 873f3615984646159e1b05178c933e18 P 8d574fb4a89d4eea947c859470d44dd4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:01.663811 16568 ts_tablet_manager.cc:1888] T 873f3615984646159e1b05178c933e18 P 8d574fb4a89d4eea947c859470d44dd4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:00:01.663879 16568 log.cc:1196] T 873f3615984646159e1b05178c933e18 P 8d574fb4a89d4eea947c859470d44dd4: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/873f3615984646159e1b05178c933e18
I20240503 14:00:01.664462 6373 catalog_manager.cc:4830] TS 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481): tablet 873f3615984646159e1b05178c933e18 (table rebalancer_extra_table_0 [id=0be76966adde4247b7969f0f470308fd]) successfully deleted
I20240503 14:00:01.672183 16574 raft_consensus.cc:482] T e92eaea6e9c8493690b749f162949ac4 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.672302 16574 raft_consensus.cc:504] T e92eaea6e9c8493690b749f162949ac4 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 14:00:01.672734 16574 leader_election.cc:288] T e92eaea6e9c8493690b749f162949ac4 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 14:00:01.673729 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92eaea6e9c8493690b749f162949ac4" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
W20240503 14:00:01.674414 6465 leader_election.cc:334] T e92eaea6e9c8493690b749f162949ac4 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.674851 6464 leader_election.cc:302] T e92eaea6e9c8493690b749f162949ac4 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da; no voters: 06ca8dbc004c42b99faa527a4e6f9545, 0d77752181e944698e809e325e5937e6
I20240503 14:00:01.676030 16574 raft_consensus.cc:2726] T e92eaea6e9c8493690b749f162949ac4 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:01.676317 15876 raft_consensus.cc:979] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43: : Instructing follower 5f46e19644fd4f4b825c86ba6af768da to start an election
I20240503 14:00:01.676430 15876 raft_consensus.cc:1067] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Signalling peer 5f46e19644fd4f4b825c86ba6af768da to start an election
I20240503 14:00:01.676990 6526 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b6cb5119a6db403ba3cae9fa9b121211"
dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
from {username='slave'} at 127.0.99.130:57965
I20240503 14:00:01.677152 6526 raft_consensus.cc:482] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:00:01.677224 6526 raft_consensus.cc:3037] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:01.678344 6526 raft_consensus.cc:504] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
I20240503 14:00:01.678884 6526 leader_election.cc:288] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 14:00:01.679208 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6cb5119a6db403ba3cae9fa9b121211" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 14:00:01.679348 6922 raft_consensus.cc:3037] T b6cb5119a6db403ba3cae9fa9b121211 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:01.679414 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6cb5119a6db403ba3cae9fa9b121211" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 14:00:01.679525 8947 raft_consensus.cc:3037] T b6cb5119a6db403ba3cae9fa9b121211 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:01.680426 6922 raft_consensus.cc:2454] T b6cb5119a6db403ba3cae9fa9b121211 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 2.
I20240503 14:00:01.680543 8947 raft_consensus.cc:2454] T b6cb5119a6db403ba3cae9fa9b121211 P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 2.
I20240503 14:00:01.680907 15959 raft_consensus.cc:482] T 1eec99c0bb994dd7b6ae63918125bccb P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.680915 6464 leader_election.cc:302] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 14:00:01.680996 15959 raft_consensus.cc:504] T 1eec99c0bb994dd7b6ae63918125bccb P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 14:00:01.681365 15959 leader_election.cc:288] T 1eec99c0bb994dd7b6ae63918125bccb P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 14:00:01.681514 16574 raft_consensus.cc:2781] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:01.681602 16574 raft_consensus.cc:686] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da [term 2 LEADER]: Becoming Leader. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Running, Role: LEADER
I20240503 14:00:01.681665 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eec99c0bb994dd7b6ae63918125bccb" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 14:00:01.681754 16574 consensus_queue.cc:237] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
W20240503 14:00:01.685195 6596 leader_election.cc:334] T 1eec99c0bb994dd7b6ae63918125bccb P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.685281 6596 leader_election.cc:302] T 1eec99c0bb994dd7b6ae63918125bccb P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43; no voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735
I20240503 14:00:01.685560 15959 raft_consensus.cc:2726] T 1eec99c0bb994dd7b6ae63918125bccb P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:01.685791 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6cb5119a6db403ba3cae9fa9b121211" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 14:00:01.686110 6659 raft_consensus.cc:3032] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:00:01.686203 6659 raft_consensus.cc:729] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 14:00:01.686177 6380 catalog_manager.cc:5484] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: term changed from 1 to 2, leader changed from 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) to 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129). New cstate: current_term: 2 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:01.686416 6659 consensus_queue.cc:260] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
I20240503 14:00:01.686645 6659 raft_consensus.cc:3037] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:01.688354 6659 raft_consensus.cc:2454] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 2.
I20240503 14:00:01.700771 15876 raft_consensus.cc:482] T 6ee85f4483004c1f9499203c7323706d P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
W20240503 14:00:01.700938 9146 connection.cc:535] client connection to 127.0.99.131:46365 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:00:01.700881 15876 raft_consensus.cc:504] T 6ee85f4483004c1f9499203c7323706d P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
W20240503 14:00:01.701151 8882 connection.cc:535] server connection from 127.0.99.131:54275 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:00:01.701313 15876 leader_election.cc:288] T 6ee85f4483004c1f9499203c7323706d P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 14:00:01.701624 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee85f4483004c1f9499203c7323706d" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
W20240503 14:00:01.701997 8882 connection.cc:535] client connection to 127.0.99.131:46365 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:00:01.702023 6463 connection.cc:535] client connection to 127.0.99.131:46365 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:00:01.703023 6856 connection.cc:535] client connection to 127.0.99.131:46365 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:00:01.703035 9015 connection.cc:535] client connection to 127.0.99.131:46365 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:00:01.703675 6856 connection.cc:535] server connection from 127.0.99.131:38291 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:00:01.703704 6594 connection.cc:535] client connection to 127.0.99.131:46365 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:00:01.703867 6596 leader_election.cc:334] T 6ee85f4483004c1f9499203c7323706d P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.703985 6596 leader_election.cc:302] T 6ee85f4483004c1f9499203c7323706d P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43; no voters: 0d77752181e944698e809e325e5937e6, f58065db70b64ef8b1dc499aa6ae4460
I20240503 14:00:01.704159 15876 raft_consensus.cc:2726] T 6ee85f4483004c1f9499203c7323706d P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:01.704799 6372 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.99.131:46365 (error 108)
W20240503 14:00:01.705689 16474 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:00:01.702501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.131:46365 (local address 127.0.99.129:40161)
0503 14:00:01.702657 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0503 14:00:01.702668 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 14:00:01.702771 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:00:01.705177 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":82}
W20240503 14:00:01.705690 16473 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:00:01.701491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.131:46365 (local address 127.0.99.135:36997)
0503 14:00:01.701668 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0503 14:00:01.701686 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0503 14:00:01.701787 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:00:01.705163 (+ 3376us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":42}
W20240503 14:00:01.705840 6463 consensus_peers.cc:485] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.706939 16462 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:00:01.703523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.131:46365 (local address 127.0.99.132:54249)
0503 14:00:01.703658 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0503 14:00:01.703670 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 14:00:01.703771 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:00:01.706575 (+ 2804us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":47}
W20240503 14:00:01.707630 16469 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:00:01.703593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.131:46365 (local address 127.0.99.134:38301)
0503 14:00:01.703810 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0503 14:00:01.703823 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0503 14:00:01.703930 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:00:01.707265 (+ 3335us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":123}
W20240503 14:00:01.707818 9015 consensus_peers.cc:485] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.707875 9015 consensus_peers.cc:485] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.708333 9015 consensus_peers.cc:485] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.708559 16464 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:00:01.702501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.131:46365 (local address 127.0.99.133:44857)
0503 14:00:01.702607 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0503 14:00:01.702681 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0503 14:00:01.702794 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:00:01.708220 (+ 5426us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":30}
W20240503 14:00:01.708729 8882 consensus_peers.cc:485] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.708796 8882 consensus_peers.cc:485] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.709069 16465 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:00:01.704280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.131:46365 (local address 127.0.99.130:36329)
0503 14:00:01.704368 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0503 14:00:01.704378 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 14:00:01.704444 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:00:01.708713 (+ 4269us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":23}
W20240503 14:00:01.709307 6594 consensus_peers.cc:485] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.709381 6594 consensus_peers.cc:485] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.709566 6856 consensus_peers.cc:485] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.709836 9146 consensus_peers.cc:485] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.709898 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:00:01.709936 9146 consensus_peers.cc:485] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.709978 9146 consensus_peers.cc:485] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.710026 9146 consensus_peers.cc:485] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.711010 9146 consensus_peers.cc:485] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.711478 6372 catalog_manager.cc:4573] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3856ff74b3bf4f608e3c6cdd06ce144e: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:01.717355 16610 raft_consensus.cc:482] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.717458 16610 raft_consensus.cc:504] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 14:00:01.717847 16610 leader_election.cc:288] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
W20240503 14:00:01.719732 9017 leader_election.cc:334] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.720031 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60d453dd9c04ffaa6af6a6e233c7ef2" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
W20240503 14:00:01.720095 6594 consensus_peers.cc:485] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:01.720468 9017 leader_election.cc:302] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4; no voters: 0d77752181e944698e809e325e5937e6, 2145d15ab2d74c0287839266f1411a43
I20240503 14:00:01.720669 16610 raft_consensus.cc:2726] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:01.724293 14390 raft_consensus.cc:482] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.724417 14390 raft_consensus.cc:504] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
I20240503 14:00:01.724853 14390 leader_election.cc:288] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 14:00:01.725205 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe53d65ab4d482aad118d6e3bd5533c" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
W20240503 14:00:01.727994 8884 leader_election.cc:334] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.728072 8884 leader_election.cc:302] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545; no voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735
I20240503 14:00:01.728202 14390 raft_consensus.cc:2726] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:01.730942 6594 consensus_peers.cc:485] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:01.735141 9205 consensus_queue.cc:237] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
W20240503 14:00:01.737183 9146 consensus_peers.cc:485] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.737286 9148 consensus_peers.cc:485] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.737430 9147 consensus_peers.cc:485] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): Couldn't send request to peer 06ca8dbc004c42b99faa527a4e6f9545. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ff9b89e4e3247c59ae48ecfeb72f56b. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.752097 9146 consensus_peers.cc:485] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.756683 6856 consensus_peers.cc:485] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:01.760462 12063 raft_consensus.cc:482] T abe859639139483aa1e57ee6c9722b87 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.760581 12063 raft_consensus.cc:504] T abe859639139483aa1e57ee6c9722b87 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 14:00:01.760934 12063 leader_election.cc:288] T abe859639139483aa1e57ee6c9722b87 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 14:00:01.761438 6526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abe859639139483aa1e57ee6c9722b87" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
W20240503 14:00:01.761934 9148 leader_election.cc:334] T abe859639139483aa1e57ee6c9722b87 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.762007 9148 leader_election.cc:302] T abe859639139483aa1e57ee6c9722b87 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab1d21935d6049888eb0ceccc3c9d735; no voters: 0d77752181e944698e809e325e5937e6, 5f46e19644fd4f4b825c86ba6af768da
I20240503 14:00:01.762136 12063 raft_consensus.cc:2726] T abe859639139483aa1e57ee6c9722b87 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:01.772766 8882 consensus_peers.cc:485] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:01.777629 15416 raft_consensus.cc:482] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.777755 15416 raft_consensus.cc:504] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 14:00:01.778239 15416 leader_election.cc:288] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 14:00:01.778638 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e29a1f8d9c5c40aaa4023dd3d4204fb5" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
W20240503 14:00:01.779175 6465 leader_election.cc:334] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.779242 6465 leader_election.cc:302] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da; no voters: 06ca8dbc004c42b99faa527a4e6f9545, 0d77752181e944698e809e325e5937e6
I20240503 14:00:01.779526 15416 raft_consensus.cc:2726] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:01.795540 6463 consensus_peers.cc:485] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.797111 6856 consensus_peers.cc:485] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.804667 6856 consensus_peers.cc:485] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:01.806713 15876 raft_consensus.cc:482] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.806836 15876 raft_consensus.cc:504] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 14:00:01.807303 15876 leader_election.cc:288] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 14:00:01.807579 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60d453dd9c04ffaa6af6a6e233c7ef2" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 14:00:01.807709 9081 raft_consensus.cc:2454] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 14:00:01.807979 6595 leader_election.cc:302] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 14:00:01.808213 15876 raft_consensus.cc:2781] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:01.808267 15876 raft_consensus.cc:482] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d77752181e944698e809e325e5937e6)
W20240503 14:00:01.808300 6596 leader_election.cc:334] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.808306 15876 raft_consensus.cc:3037] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:01.809252 15876 raft_consensus.cc:504] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 14:00:01.809545 15876 leader_election.cc:288] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 14:00:01.809875 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60d453dd9c04ffaa6af6a6e233c7ef2" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 14:00:01.810029 9081 raft_consensus.cc:3037] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:00:01.810442 6596 leader_election.cc:334] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.811134 9081 raft_consensus.cc:2454] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 2.
I20240503 14:00:01.811462 6595 leader_election.cc:302] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 8d574fb4a89d4eea947c859470d44dd4; no voters: 0d77752181e944698e809e325e5937e6
I20240503 14:00:01.811658 15876 raft_consensus.cc:2781] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:01.812147 15876 raft_consensus.cc:686] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 [term 2 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 14:00:01.812297 15876 consensus_queue.cc:237] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 14:00:01.812369 16009 raft_consensus.cc:482] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.812453 16009 raft_consensus.cc:504] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 14:00:01.812765 16009 leader_election.cc:288] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 14:00:01.813133 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b397d0460c94438a773be184312b60f" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
W20240503 14:00:01.814371 9017 leader_election.cc:334] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.814445 9017 leader_election.cc:302] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4; no voters: 0d77752181e944698e809e325e5937e6, f58065db70b64ef8b1dc499aa6ae4460
I20240503 14:00:01.814563 16009 raft_consensus.cc:2726] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:01.815886 6387 catalog_manager.cc:5484] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 1 to 2, leader changed from 0d77752181e944698e809e325e5937e6 (127.0.99.136) to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } }
W20240503 14:00:01.816318 6594 consensus_peers.cc:485] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.821975 9146 consensus_peers.cc:485] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.826488 6594 consensus_peers.cc:485] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:01.845990 16610 raft_consensus.cc:482] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.846117 16610 raft_consensus.cc:504] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 14:00:01.846576 16610 leader_election.cc:288] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 14:00:01.846925 9210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d38e1d60dd42eabf7191598ded8cd5" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
W20240503 14:00:01.847793 9017 leader_election.cc:334] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.847860 9017 leader_election.cc:302] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4; no voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735
I20240503 14:00:01.847980 16610 raft_consensus.cc:2726] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:01.853796 9146 consensus_peers.cc:485] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:01.855369 15416 raft_consensus.cc:482] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.855474 15416 raft_consensus.cc:504] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 14:00:01.855901 15416 leader_election.cc:288] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 14:00:01.856434 12063 raft_consensus.cc:482] T da57af787aef40628e2c043023763f02 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.856528 12063 raft_consensus.cc:504] T da57af787aef40628e2c043023763f02 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
W20240503 14:00:01.856870 6463 leader_election.cc:334] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:01.856931 12063 leader_election.cc:288] T da57af787aef40628e2c043023763f02 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 14:00:01.857451 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da57af787aef40628e2c043023763f02" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
W20240503 14:00:01.857815 6465 leader_election.cc:334] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.857879 6465 leader_election.cc:302] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
W20240503 14:00:01.857875 9148 leader_election.cc:334] T da57af787aef40628e2c043023763f02 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.858028 15416 raft_consensus.cc:2726] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:01.858285 9147 leader_election.cc:302] T da57af787aef40628e2c043023763f02 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab1d21935d6049888eb0ceccc3c9d735; no voters: 06ca8dbc004c42b99faa527a4e6f9545, 0d77752181e944698e809e325e5937e6
I20240503 14:00:01.858551 12063 raft_consensus.cc:2726] T da57af787aef40628e2c043023763f02 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:01.863400 15416 raft_consensus.cc:482] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.863484 15416 raft_consensus.cc:504] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 14:00:01.863802 15416 leader_election.cc:288] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 14:00:01.863826 16610 raft_consensus.cc:482] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.863924 16610 raft_consensus.cc:504] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 14:00:01.864208 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d9906a118744dd6a78d50e52f1961e2" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 14:00:01.864292 16610 leader_election.cc:288] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 14:00:01.864301 9080 raft_consensus.cc:2454] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 14:00:01.864569 6464 leader_election.cc:302] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 14:00:01.864629 6526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d9906a118744dd6a78d50e52f1961e2" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 14:00:01.864742 6526 raft_consensus.cc:2454] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 14:00:01.864815 15416 raft_consensus.cc:2781] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:01.864884 15416 raft_consensus.cc:482] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.864930 15416 raft_consensus.cc:3037] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:01.865058 9016 leader_election.cc:302] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 14:00:01.865221 16610 raft_consensus.cc:2781] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:01.865270 16610 raft_consensus.cc:482] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.865300 16610 raft_consensus.cc:3037] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:00:01.865562 9017 leader_election.cc:334] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.866011 15416 raft_consensus.cc:504] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 14:00:01.866132 16610 raft_consensus.cc:504] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 14:00:01.866262 15416 leader_election.cc:288] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 14:00:01.866438 16610 leader_election.cc:288] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 14:00:01.866811 6526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d9906a118744dd6a78d50e52f1961e2" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 14:00:01.866935 6526 raft_consensus.cc:2379] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8d574fb4a89d4eea947c859470d44dd4 in current term 2: Already voted for candidate 5f46e19644fd4f4b825c86ba6af768da in this term.
I20240503 14:00:01.866564 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d9906a118744dd6a78d50e52f1961e2" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 14:00:01.867116 9080 raft_consensus.cc:2379] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f46e19644fd4f4b825c86ba6af768da in current term 2: Already voted for candidate 8d574fb4a89d4eea947c859470d44dd4 in this term.
W20240503 14:00:01.867305 6465 leader_election.cc:334] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:01.867542 9017 leader_election.cc:334] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.867609 9017 leader_election.cc:302] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4; no voters: 0d77752181e944698e809e325e5937e6, 5f46e19644fd4f4b825c86ba6af768da
I20240503 14:00:01.867751 16610 raft_consensus.cc:2726] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240503 14:00:01.868072 6465 leader_election.cc:334] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.868137 6465 leader_election.cc:302] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da; no voters: 0d77752181e944698e809e325e5937e6, 8d574fb4a89d4eea947c859470d44dd4
I20240503 14:00:01.868434 15416 raft_consensus.cc:2726] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240503 14:00:01.869087 8882 consensus_peers.cc:485] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.881723 6856 consensus_peers.cc:485] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:01.892338 8882 consensus_peers.cc:485] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:01.905030 12063 raft_consensus.cc:482] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.905218 12063 raft_consensus.cc:504] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
I20240503 14:00:01.905651 12063 leader_election.cc:288] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 14:00:01.906057 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe53d65ab4d482aad118d6e3bd5533c" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 14:00:01.906239 8947 raft_consensus.cc:2454] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 14:00:01.906504 9147 leader_election.cc:302] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 14:00:01.906720 12063 raft_consensus.cc:2781] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:01.906786 12063 raft_consensus.cc:482] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.906848 12063 raft_consensus.cc:3037] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:01.906922 16610 raft_consensus.cc:482] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.907068 16610 raft_consensus.cc:504] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
W20240503 14:00:01.907114 9148 leader_election.cc:334] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.907753 9210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ddf3f4cf2884b8785b216a59a5b40f9" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 14:00:01.907642 16610 leader_election.cc:288] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 14:00:01.908046 12063 raft_consensus.cc:504] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
I20240503 14:00:01.908406 12063 leader_election.cc:288] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 14:00:01.908859 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe53d65ab4d482aad118d6e3bd5533c" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 14:00:01.908977 8947 raft_consensus.cc:3037] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:01.910081 8947 raft_consensus.cc:2454] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 2.
W20240503 14:00:01.910502 9148 leader_election.cc:334] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.910640 9147 leader_election.cc:302] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters: 0d77752181e944698e809e325e5937e6
I20240503 14:00:01.910784 12063 raft_consensus.cc:2781] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:01.910938 12063 raft_consensus.cc:686] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [term 2 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 14:00:01.911087 12063 consensus_queue.cc:237] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
W20240503 14:00:01.911972 9017 leader_election.cc:334] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.912045 9017 leader_election.cc:302] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4; no voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735
I20240503 14:00:01.912189 16610 raft_consensus.cc:2726] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:01.913465 6387 catalog_manager.cc:5484] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 1 to 2, leader changed from 0d77752181e944698e809e325e5937e6 (127.0.99.136) to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:01.916136 12063 raft_consensus.cc:482] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.916214 12063 raft_consensus.cc:504] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 14:00:01.916541 12063 leader_election.cc:288] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 14:00:01.916903 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ddf3f4cf2884b8785b216a59a5b40f9" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 14:00:01.917039 9080 raft_consensus.cc:2454] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 14:00:01.917284 9147 leader_election.cc:302] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 14:00:01.917450 12063 raft_consensus.cc:2781] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240503 14:00:01.917482 9148 leader_election.cc:334] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.917574 12063 raft_consensus.cc:482] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.917629 12063 raft_consensus.cc:3037] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:01.918402 12063 raft_consensus.cc:504] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 14:00:01.918787 12063 leader_election.cc:288] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 14:00:01.918984 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ddf3f4cf2884b8785b216a59a5b40f9" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 14:00:01.919116 9080 raft_consensus.cc:3037] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:00:01.919951 9148 leader_election.cc:334] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.920346 9080 raft_consensus.cc:2454] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 2.
I20240503 14:00:01.920598 9147 leader_election.cc:302] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, ab1d21935d6049888eb0ceccc3c9d735; no voters: 0d77752181e944698e809e325e5937e6
I20240503 14:00:01.920747 12063 raft_consensus.cc:2781] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:01.920867 12063 raft_consensus.cc:686] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 14:00:01.921074 12063 consensus_queue.cc:237] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
W20240503 14:00:01.921991 9146 consensus_peers.cc:485] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:01.923641 6387 catalog_manager.cc:5484] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 1 to 2, leader changed from 0d77752181e944698e809e325e5937e6 (127.0.99.136) to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:01.942111 16009 raft_consensus.cc:482] T e76ded33d780455d963c51a253cbd0e0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.942238 16009 raft_consensus.cc:504] T e76ded33d780455d963c51a253cbd0e0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 14:00:01.942606 16009 leader_election.cc:288] T e76ded33d780455d963c51a253cbd0e0 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 14:00:01.942909 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e76ded33d780455d963c51a253cbd0e0" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
W20240503 14:00:01.943872 9017 leader_election.cc:334] T e76ded33d780455d963c51a253cbd0e0 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.943953 9017 leader_election.cc:302] T e76ded33d780455d963c51a253cbd0e0 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4; no voters: 0d77752181e944698e809e325e5937e6, 2145d15ab2d74c0287839266f1411a43
I20240503 14:00:01.944113 16009 raft_consensus.cc:2726] T e76ded33d780455d963c51a253cbd0e0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:01.948144 14177 raft_consensus.cc:482] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.948261 14177 raft_consensus.cc:504] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 14:00:01.948621 14177 leader_election.cc:288] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 14:00:01.948886 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e76ded33d780455d963c51a253cbd0e0" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 14:00:01.949034 9081 raft_consensus.cc:2454] T e76ded33d780455d963c51a253cbd0e0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 14:00:01.949321 6595 leader_election.cc:302] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 14:00:01.949522 14177 raft_consensus.cc:2781] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:01.949575 14177 raft_consensus.cc:482] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.949630 14177 raft_consensus.cc:3037] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:00:01.949661 6596 leader_election.cc:334] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.950462 14177 raft_consensus.cc:504] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 14:00:01.950775 14177 leader_election.cc:288] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 14:00:01.951345 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e76ded33d780455d963c51a253cbd0e0" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 14:00:01.951486 9081 raft_consensus.cc:3037] T e76ded33d780455d963c51a253cbd0e0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:00:01.952212 6596 leader_election.cc:334] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.952430 9081 raft_consensus.cc:2454] T e76ded33d780455d963c51a253cbd0e0 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 2.
I20240503 14:00:01.952718 6595 leader_election.cc:302] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 8d574fb4a89d4eea947c859470d44dd4; no voters: 0d77752181e944698e809e325e5937e6
I20240503 14:00:01.952880 14177 raft_consensus.cc:2781] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Leader election won for term 2
W20240503 14:00:01.952937 6594 consensus_peers.cc:485] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:01.952975 14177 raft_consensus.cc:686] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 [term 2 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 14:00:01.953126 14177 consensus_queue.cc:237] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } }
I20240503 14:00:01.955667 6387 catalog_manager.cc:5484] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 1 to 2, leader changed from 0d77752181e944698e809e325e5937e6 (127.0.99.136) to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:01.972375 14390 raft_consensus.cc:482] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.972482 14390 raft_consensus.cc:504] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 14:00:01.972882 14390 leader_election.cc:288] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 14:00:01.973028 15111 raft_consensus.cc:482] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.973172 15111 raft_consensus.cc:504] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 14:00:01.973245 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da57af787aef40628e2c043023763f02" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 14:00:01.973395 9205 raft_consensus.cc:2454] T da57af787aef40628e2c043023763f02 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 14:00:01.973563 15111 leader_election.cc:288] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 14:00:01.973680 8883 leader_election.cc:302] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 14:00:01.973898 14390 raft_consensus.cc:2781] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:01.973963 14390 raft_consensus.cc:482] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.974006 14390 raft_consensus.cc:3037] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:00:01.974452 8884 leader_election.cc:334] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.974577 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee85f4483004c1f9499203c7323706d" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 14:00:01.974692 6659 raft_consensus.cc:2454] T 6ee85f4483004c1f9499203c7323706d P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
W20240503 14:00:01.974735 6858 leader_election.cc:334] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.974956 6858 leader_election.cc:302] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6
I20240503 14:00:01.975143 15111 raft_consensus.cc:2781] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:01.975206 15111 raft_consensus.cc:482] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.975234 15111 raft_consensus.cc:3037] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:01.975225 14390 raft_consensus.cc:504] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 14:00:01.975569 14390 leader_election.cc:288] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 14:00:01.975775 12063 raft_consensus.cc:482] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.975868 12063 raft_consensus.cc:504] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 14:00:01.976249 12063 leader_election.cc:288] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 14:00:01.976351 15111 raft_consensus.cc:504] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 14:00:01.976689 15111 leader_election.cc:288] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 14:00:01.977306 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da57af787aef40628e2c043023763f02" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 14:00:01.977411 9205 raft_consensus.cc:3037] T da57af787aef40628e2c043023763f02 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:01.977583 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee85f4483004c1f9499203c7323706d" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 14:00:01.977702 6659 raft_consensus.cc:3037] T 6ee85f4483004c1f9499203c7323706d P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:00:01.977941 6858 leader_election.cc:334] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.978387 9205 raft_consensus.cc:2454] T da57af787aef40628e2c043023763f02 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 2.
I20240503 14:00:01.978683 6659 raft_consensus.cc:2454] T 6ee85f4483004c1f9499203c7323706d P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 2.
I20240503 14:00:01.978844 8883 leader_election.cc:302] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 14:00:01.978866 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d38e1d60dd42eabf7191598ded8cd5" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
W20240503 14:00:01.978986 9148 leader_election.cc:334] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.978996 9081 raft_consensus.cc:2454] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 14:00:01.979131 14390 raft_consensus.cc:2781] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:01.979259 14390 raft_consensus.cc:686] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 14:00:01.979312 6858 leader_election.cc:302] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6
I20240503 14:00:01.979444 15111 raft_consensus.cc:2781] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:01.979449 14390 consensus_queue.cc:237] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 14:00:01.979521 15111 raft_consensus.cc:686] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [term 2 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 14:00:01.979646 15111 consensus_queue.cc:237] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
W20240503 14:00:01.979151 8884 leader_election.cc:334] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.981019 9147 leader_election.cc:302] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, ab1d21935d6049888eb0ceccc3c9d735; no voters: 0d77752181e944698e809e325e5937e6
I20240503 14:00:01.981191 12063 raft_consensus.cc:2781] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:01.981252 12063 raft_consensus.cc:482] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:01.981294 12063 raft_consensus.cc:3037] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:01.982029 6387 catalog_manager.cc:5484] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 1 to 2, leader changed from 0d77752181e944698e809e325e5937e6 (127.0.99.136) to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:01.982286 12063 raft_consensus.cc:504] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 14:00:01.982295 6380 catalog_manager.cc:5484] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 1 to 2, leader changed from 0d77752181e944698e809e325e5937e6 (127.0.99.136) to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 2 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } }
I20240503 14:00:01.982662 12063 leader_election.cc:288] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 14:00:01.982995 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d38e1d60dd42eabf7191598ded8cd5" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 14:00:01.983122 9081 raft_consensus.cc:3037] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:00:01.983569 9148 leader_election.cc:334] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:01.984163 9081 raft_consensus.cc:2454] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 2.
I20240503 14:00:01.984494 9147 leader_election.cc:302] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, ab1d21935d6049888eb0ceccc3c9d735; no voters: 0d77752181e944698e809e325e5937e6
I20240503 14:00:01.984699 12063 raft_consensus.cc:2781] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:01.984797 12063 raft_consensus.cc:686] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 14:00:01.984911 12063 consensus_queue.cc:237] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 14:00:01.986995 6387 catalog_manager.cc:5484] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 1 to 2, leader changed from 0d77752181e944698e809e325e5937e6 (127.0.99.136) to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 14:00:01.992302 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:02.001525 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:02.003769 14390 raft_consensus.cc:482] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.003873 14390 raft_consensus.cc:504] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 14:00:02.004272 14390 leader_election.cc:288] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 14:00:02.004601 6526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92eaea6e9c8493690b749f162949ac4" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 14:00:02.004748 6526 raft_consensus.cc:2454] T e92eaea6e9c8493690b749f162949ac4 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 14:00:02.005055 8883 leader_election.cc:302] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 5f46e19644fd4f4b825c86ba6af768da; no voters:
W20240503 14:00:02.005321 8884 leader_election.cc:334] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:02.005347 14390 raft_consensus.cc:2781] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:02.005398 14390 raft_consensus.cc:482] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.005432 14390 raft_consensus.cc:3037] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.006310 14390 raft_consensus.cc:504] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 14:00:02.006732 14390 leader_election.cc:288] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 14:00:02.007057 6526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92eaea6e9c8493690b749f162949ac4" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 14:00:02.007176 6526 raft_consensus.cc:3037] T e92eaea6e9c8493690b749f162949ac4 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:00:02.007764 8884 leader_election.cc:334] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:02.008145 6526 raft_consensus.cc:2454] T e92eaea6e9c8493690b749f162949ac4 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 2.
I20240503 14:00:02.008409 8883 leader_election.cc:302] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 5f46e19644fd4f4b825c86ba6af768da; no voters: 0d77752181e944698e809e325e5937e6
I20240503 14:00:02.008591 14390 raft_consensus.cc:2781] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:02.008675 14390 raft_consensus.cc:686] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 14:00:02.008822 14390 consensus_queue.cc:237] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 14:00:02.012051 6380 catalog_manager.cc:5484] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 1 to 2, leader changed from 0d77752181e944698e809e325e5937e6 (127.0.99.136) to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 2 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:02.027125 14177 raft_consensus.cc:482] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.027248 14177 raft_consensus.cc:504] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 14:00:02.027702 14177 leader_election.cc:288] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 14:00:02.028033 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442c955ab10f4ecbafdc9bf89d8fb39e" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
W20240503 14:00:02.028751 6596 leader_election.cc:334] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:02.028822 6596 leader_election.cc:302] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43; no voters: 0d77752181e944698e809e325e5937e6, f58065db70b64ef8b1dc499aa6ae4460
I20240503 14:00:02.028954 14177 raft_consensus.cc:2726] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:02.032104 12748 raft_consensus.cc:482] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.032207 12748 raft_consensus.cc:504] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 14:00:02.032567 12748 leader_election.cc:288] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 14:00:02.032927 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b397d0460c94438a773be184312b60f" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 14:00:02.033071 9081 raft_consensus.cc:2454] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 14:00:02.033496 6857 leader_election.cc:302] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 14:00:02.033845 15927 raft_consensus.cc:2781] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:02.033914 15927 raft_consensus.cc:482] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.033955 15927 raft_consensus.cc:3037] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:00:02.034250 6858 leader_election.cc:334] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:02.034931 15927 raft_consensus.cc:504] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 14:00:02.035203 15927 leader_election.cc:288] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 election: Requested vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 14:00:02.036026 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b397d0460c94438a773be184312b60f" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 14:00:02.036146 9081 raft_consensus.cc:3037] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.036934 16096 raft_consensus.cc:482] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.037034 16096 raft_consensus.cc:504] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 14:00:02.037262 9081 raft_consensus.cc:2454] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 2.
I20240503 14:00:02.037472 16096 leader_election.cc:288] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 14:00:02.037544 6857 leader_election.cc:302] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 14:00:02.037693 12748 raft_consensus.cc:2781] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:02.037768 12748 raft_consensus.cc:686] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 [term 2 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 14:00:02.037916 12748 consensus_queue.cc:237] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
W20240503 14:00:02.038681 9148 leader_election.cc:334] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:02.038868 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a914c22ceff44fc685aad0fdcd04d844" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 14:00:02.038975 6922 raft_consensus.cc:2454] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
W20240503 14:00:02.039119 6858 leader_election.cc:334] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:02.039223 9147 leader_election.cc:302] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab1d21935d6049888eb0ceccc3c9d735, f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6
I20240503 14:00:02.039300 12748 raft_consensus.cc:482] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.039381 12748 raft_consensus.cc:504] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 14:00:02.039685 12748 leader_election.cc:288] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 14:00:02.039830 16096 raft_consensus.cc:2781] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:02.039880 16096 raft_consensus.cc:482] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.039917 16096 raft_consensus.cc:3037] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.040234 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a914c22ceff44fc685aad0fdcd04d844" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 14:00:02.040900 6380 catalog_manager.cc:5484] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 1 to 2, leader changed from 0d77752181e944698e809e325e5937e6 (127.0.99.136) to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:02.041042 16096 raft_consensus.cc:504] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 14:00:02.041265 9205 raft_consensus.cc:2379] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f58065db70b64ef8b1dc499aa6ae4460 in current term 2: Already voted for candidate ab1d21935d6049888eb0ceccc3c9d735 in this term.
I20240503 14:00:02.041363 16096 leader_election.cc:288] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
W20240503 14:00:02.042052 6858 leader_election.cc:334] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:02.042132 6858 leader_election.cc:302] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735
I20240503 14:00:02.042459 15927 raft_consensus.cc:3037] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.042479 15416 raft_consensus.cc:482] T 0fa4727297da40eb9f58f5daa8150e0e P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.042582 15416 raft_consensus.cc:504] T 0fa4727297da40eb9f58f5daa8150e0e P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 14:00:02.042985 15416 leader_election.cc:288] T 0fa4727297da40eb9f58f5daa8150e0e P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 14:00:02.043519 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fa4727297da40eb9f58f5daa8150e0e" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 14:00:02.043582 15927 raft_consensus.cc:2726] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:02.043256 9148 leader_election.cc:334] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:02.043991 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a914c22ceff44fc685aad0fdcd04d844" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
W20240503 14:00:02.044379 6465 leader_election.cc:334] T 0fa4727297da40eb9f58f5daa8150e0e P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:02.044456 6465 leader_election.cc:302] T 0fa4727297da40eb9f58f5daa8150e0e P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da; no voters: 0d77752181e944698e809e325e5937e6, 2145d15ab2d74c0287839266f1411a43
I20240503 14:00:02.044576 15416 raft_consensus.cc:2726] T 0fa4727297da40eb9f58f5daa8150e0e P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:02.044999 6922 raft_consensus.cc:2454] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 2.
I20240503 14:00:02.045293 9147 leader_election.cc:302] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab1d21935d6049888eb0ceccc3c9d735, f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6
I20240503 14:00:02.045486 16096 raft_consensus.cc:2781] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:02.045656 16096 raft_consensus.cc:686] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 14:00:02.045818 16096 consensus_queue.cc:237] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 14:00:02.048303 6380 catalog_manager.cc:5484] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 1 to 2, leader changed from 0d77752181e944698e809e325e5937e6 (127.0.99.136) to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:02.058890 12748 raft_consensus.cc:482] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.059012 12748 raft_consensus.cc:504] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 14:00:02.059428 12748 leader_election.cc:288] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
W20240503 14:00:02.060422 6858 leader_election.cc:334] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:02.060520 6856 leader_election.cc:334] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.060618 6856 leader_election.cc:302] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:02.060783 15927 raft_consensus.cc:2726] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:02.060775 12748 raft_consensus.cc:482] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.061020 12748 raft_consensus.cc:504] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 14:00:02.061303 12748 leader_election.cc:288] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
W20240503 14:00:02.062153 6858 leader_election.cc:334] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:02.062261 6856 leader_election.cc:334] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.062337 6856 leader_election.cc:302] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:02.062484 12748 raft_consensus.cc:2726] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:02.062633 8947 raft_consensus.cc:1261] T b6cb5119a6db403ba3cae9fa9b121211 P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:02.063001 15416 consensus_queue.cc:1035] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:02.064514 14390 raft_consensus.cc:482] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.064602 14390 raft_consensus.cc:504] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 14:00:02.064970 14390 leader_election.cc:288] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
W20240503 14:00:02.065896 8884 leader_election.cc:334] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:02.066483 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e29a1f8d9c5c40aaa4023dd3d4204fb5" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 14:00:02.066625 6527 raft_consensus.cc:2454] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 14:00:02.067252 8883 leader_election.cc:302] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 5f46e19644fd4f4b825c86ba6af768da; no voters: 0d77752181e944698e809e325e5937e6
I20240503 14:00:02.067525 14390 raft_consensus.cc:2781] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:02.067591 14390 raft_consensus.cc:482] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.067644 14390 raft_consensus.cc:3037] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.068528 14390 raft_consensus.cc:504] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 14:00:02.068889 14390 leader_election.cc:288] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 14:00:02.069218 6526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e29a1f8d9c5c40aaa4023dd3d4204fb5" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 14:00:02.069355 6526 raft_consensus.cc:3037] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:00:02.069746 8884 leader_election.cc:334] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:02.070264 6526 raft_consensus.cc:2454] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 2.
I20240503 14:00:02.070559 8883 leader_election.cc:302] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 5f46e19644fd4f4b825c86ba6af768da; no voters: 0d77752181e944698e809e325e5937e6
I20240503 14:00:02.070731 14390 raft_consensus.cc:2781] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:02.070812 14390 raft_consensus.cc:686] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 14:00:02.070955 14390 consensus_queue.cc:237] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } }
I20240503 14:00:02.073099 6380 catalog_manager.cc:5484] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 1 to 2, leader changed from 0d77752181e944698e809e325e5937e6 (127.0.99.136) to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 2 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } }
I20240503 14:00:02.094830 15099 raft_consensus.cc:482] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.094936 15099 raft_consensus.cc:504] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 14:00:02.095326 15099 leader_election.cc:288] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 14:00:02.095618 15796 raft_consensus.cc:482] T 292ed55b431847f39d832ebe61e6c5e4 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.095695 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abe859639139483aa1e57ee6c9722b87" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 14:00:02.095724 15796 raft_consensus.cc:504] T 292ed55b431847f39d832ebe61e6c5e4 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 14:00:02.095832 9205 raft_consensus.cc:2454] T abe859639139483aa1e57ee6c9722b87 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 14:00:02.096112 6464 leader_election.cc:302] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 14:00:02.096138 15796 leader_election.cc:288] T 292ed55b431847f39d832ebe61e6c5e4 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 14:00:02.096387 15099 raft_consensus.cc:2781] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:02.096460 15099 raft_consensus.cc:482] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.096501 15099 raft_consensus.cc:3037] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.096534 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ed55b431847f39d832ebe61e6c5e4" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 14:00:02.097573 15099 raft_consensus.cc:504] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 14:00:02.097925 15099 leader_election.cc:288] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 14:00:02.098320 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abe859639139483aa1e57ee6c9722b87" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 14:00:02.098461 9205 raft_consensus.cc:3037] T abe859639139483aa1e57ee6c9722b87 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:00:02.098781 6465 leader_election.cc:334] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:02.099622 9205 raft_consensus.cc:2454] T abe859639139483aa1e57ee6c9722b87 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 2.
W20240503 14:00:02.099927 6465 leader_election.cc:334] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:02.099995 6465 leader_election.cc:302] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, ab1d21935d6049888eb0ceccc3c9d735; no voters: 0d77752181e944698e809e325e5937e6
I20240503 14:00:02.100242 15099 raft_consensus.cc:2781] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Leader election won for term 2
W20240503 14:00:02.100251 9017 leader_election.cc:334] T 292ed55b431847f39d832ebe61e6c5e4 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:02.100314 9017 leader_election.cc:302] T 292ed55b431847f39d832ebe61e6c5e4 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4; no voters: 0d77752181e944698e809e325e5937e6, f58065db70b64ef8b1dc499aa6ae4460
I20240503 14:00:02.100332 15099 raft_consensus.cc:686] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da [term 2 LEADER]: Becoming Leader. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Running, Role: LEADER
I20240503 14:00:02.100488 15099 consensus_queue.cc:237] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 14:00:02.100565 15796 raft_consensus.cc:2726] T 292ed55b431847f39d832ebe61e6c5e4 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:02.101274 16096 raft_consensus.cc:482] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.101378 16096 raft_consensus.cc:504] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 14:00:02.101737 16096 leader_election.cc:288] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 14:00:02.102581 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eec99c0bb994dd7b6ae63918125bccb" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 14:00:02.102720 6659 raft_consensus.cc:2454] T 1eec99c0bb994dd7b6ae63918125bccb P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
W20240503 14:00:02.102972 9148 leader_election.cc:334] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:02.103147 9148 leader_election.cc:302] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, ab1d21935d6049888eb0ceccc3c9d735; no voters: 0d77752181e944698e809e325e5937e6
I20240503 14:00:02.103559 16096 raft_consensus.cc:2781] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:02.103623 16096 raft_consensus.cc:482] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.103663 16096 raft_consensus.cc:3037] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.103998 6380 catalog_manager.cc:5484] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: term changed from 1 to 2, leader changed from 0d77752181e944698e809e325e5937e6 (127.0.99.136) to 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129). New cstate: current_term: 2 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } }
I20240503 14:00:02.104700 16096 raft_consensus.cc:504] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 14:00:02.105167 16096 leader_election.cc:288] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 14:00:02.105917 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eec99c0bb994dd7b6ae63918125bccb" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 14:00:02.106036 6659 raft_consensus.cc:3037] T 1eec99c0bb994dd7b6ae63918125bccb P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:00:02.107102 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:02.107215 6659 raft_consensus.cc:2454] T 1eec99c0bb994dd7b6ae63918125bccb P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 2.
I20240503 14:00:02.107726 9148 leader_election.cc:302] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 14:00:02.107882 16096 raft_consensus.cc:2781] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Leader election won for term 2
W20240503 14:00:02.107890 9148 leader_election.cc:334] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:02.107976 16096 raft_consensus.cc:686] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [term 2 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 14:00:02.108105 16096 consensus_queue.cc:237] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 14:00:02.110916 6380 catalog_manager.cc:5484] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 1 to 2, leader changed from 0d77752181e944698e809e325e5937e6 (127.0.99.136) to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:02.114017 6922 raft_consensus.cc:1261] T b6cb5119a6db403ba3cae9fa9b121211 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:02.114554 15416 consensus_queue.cc:1035] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:02.117019 6659 raft_consensus.cc:1261] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:02.117455 15416 consensus_queue.cc:1035] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:02.125780 16767 ts_tablet_manager.cc:902] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 06ca8dbc004c42b99faa527a4e6f9545: Initiating tablet copy from peer ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 14:00:02.125936 6526 consensus_queue.cc:237] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
I20240503 14:00:02.126505 16767 tablet_copy_client.cc:308] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:34091
I20240503 14:00:02.126940 9231 tablet_copy_service.cc:140] P ab1d21935d6049888eb0ceccc3c9d735: Received BeginTabletCopySession request for tablet 9ff9b89e4e3247c59ae48ecfeb72f56b from peer 06ca8dbc004c42b99faa527a4e6f9545 ({username='slave'} at 127.0.99.133:51367)
I20240503 14:00:02.127069 9231 tablet_copy_service.cc:161] P ab1d21935d6049888eb0ceccc3c9d735: Beginning new tablet copy session on tablet 9ff9b89e4e3247c59ae48ecfeb72f56b from peer 06ca8dbc004c42b99faa527a4e6f9545 at {username='slave'} at 127.0.99.133:51367: session id = 06ca8dbc004c42b99faa527a4e6f9545-9ff9b89e4e3247c59ae48ecfeb72f56b
I20240503 14:00:02.127378 6922 raft_consensus.cc:1261] T b6cb5119a6db403ba3cae9fa9b121211 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:00:02.127552 8947 raft_consensus.cc:1261] T b6cb5119a6db403ba3cae9fa9b121211 P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:00:02.127993 16574 consensus_queue.cc:1035] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:02.128206 9231 tablet_copy_source_session.cc:205] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:02.128296 16574 consensus_queue.cc:1035] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:02.130491 16767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ff9b89e4e3247c59ae48ecfeb72f56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:02.130477 15416 raft_consensus.cc:2932] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 14:00:02.132441 6373 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b6cb5119a6db403ba3cae9fa9b121211 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:02.132906 6920 raft_consensus.cc:2932] T b6cb5119a6db403ba3cae9fa9b121211 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 14:00:02.133383 16767 tablet_copy_client.cc:783] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:02.133527 16767 tablet_copy_client.cc:647] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:02.134286 6380 catalog_manager.cc:5484] T b6cb5119a6db403ba3cae9fa9b121211 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: config changed from index 5 to 7, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:02.137228 16767 tablet_copy_client.cc:523] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:02.138096 6639 tablet_service.cc:1514] Processing DeleteTablet for tablet b6cb5119a6db403ba3cae9fa9b121211 with delete_type TABLET_DATA_TOMBSTONED (TS 2145d15ab2d74c0287839266f1411a43 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37976
I20240503 14:00:02.137619 8947 raft_consensus.cc:2932] T b6cb5119a6db403ba3cae9fa9b121211 P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 2145d15ab2d74c0287839266f1411a43 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 14:00:02.142877 14177 raft_consensus.cc:482] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.142983 14177 raft_consensus.cc:504] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 14:00:02.143384 14177 leader_election.cc:288] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 14:00:02.144229 6526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fa4727297da40eb9f58f5daa8150e0e" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 14:00:02.144376 6526 raft_consensus.cc:2454] T 0fa4727297da40eb9f58f5daa8150e0e P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 14:00:02.144764 6595 leader_election.cc:302] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 14:00:02.144935 14177 raft_consensus.cc:2781] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:02.144994 14177 raft_consensus.cc:482] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.145037 16767 tablet_bootstrap.cc:492] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
I20240503 14:00:02.145044 14177 raft_consensus.cc:3037] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:00:02.145157 6596 leader_election.cc:334] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:02.145917 14177 raft_consensus.cc:504] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 14:00:02.146220 14177 leader_election.cc:288] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 14:00:02.146482 6526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fa4727297da40eb9f58f5daa8150e0e" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 14:00:02.146605 6526 raft_consensus.cc:3037] T 0fa4727297da40eb9f58f5daa8150e0e P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:00:02.147231 6596 leader_election.cc:334] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:02.147320 16772 tablet_replica.cc:330] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43: stopping tablet replica
I20240503 14:00:02.147534 16772 raft_consensus.cc:2229] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:02.147655 16772 raft_consensus.cc:2258] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:02.147754 6526 raft_consensus.cc:2454] T 0fa4727297da40eb9f58f5daa8150e0e P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 2.
I20240503 14:00:02.148041 6595 leader_election.cc:302] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 5f46e19644fd4f4b825c86ba6af768da; no voters: 0d77752181e944698e809e325e5937e6
I20240503 14:00:02.148236 14177 raft_consensus.cc:2781] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:02.148316 14177 raft_consensus.cc:686] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 [term 2 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 14:00:02.148466 14177 consensus_queue.cc:237] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 14:00:02.148792 16767 tablet_bootstrap.cc:492] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:00:02.149121 16767 tablet_bootstrap.cc:492] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap complete.
I20240503 14:00:02.149287 16767 ts_tablet_manager.cc:1367] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:00:02.149681 16767 raft_consensus.cc:348] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } }
I20240503 14:00:02.149880 16767 raft_consensus.cc:729] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: LEARNER
I20240503 14:00:02.149989 16767 consensus_queue.cc:260] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 14:00:02.150206 14177 raft_consensus.cc:482] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.150277 14177 raft_consensus.cc:504] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 14:00:02.151351 12748 raft_consensus.cc:482] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.151206 6379 catalog_manager.cc:5484] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 1 to 2, leader changed from 0d77752181e944698e809e325e5937e6 (127.0.99.136) to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:02.151451 12748 raft_consensus.cc:504] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
W20240503 14:00:02.151686 6596 leader_election.cc:334] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:02.151751 6594 leader_election.cc:334] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.151799 6594 leader_election.cc:302] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:02.151813 12748 leader_election.cc:288] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 14:00:02.151921 15959 raft_consensus.cc:2726] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:02.152313 16767 ts_tablet_manager.cc:1398] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20240503 14:00:02.152896 6858 leader_election.cc:334] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:02.154593 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442c955ab10f4ecbafdc9bf89d8fb39e" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 14:00:02.154717 6659 raft_consensus.cc:2454] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 14:00:02.155175 6858 leader_election.cc:302] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6
I20240503 14:00:02.155265 16772 ts_tablet_manager.cc:1875] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:02.155886 12748 raft_consensus.cc:2781] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:02.155948 12748 raft_consensus.cc:482] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.156001 12748 raft_consensus.cc:3037] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.156781 12748 raft_consensus.cc:504] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 14:00:02.157075 12748 leader_election.cc:288] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 14:00:02.157547 16772 ts_tablet_manager.cc:1888] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:00:02.157629 16772 log.cc:1196] T b6cb5119a6db403ba3cae9fa9b121211 P 2145d15ab2d74c0287839266f1411a43: Deleting WAL directory at /tmp/dist-test-taskSuJMqo/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714744715866888-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/b6cb5119a6db403ba3cae9fa9b121211
I20240503 14:00:02.157723 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442c955ab10f4ecbafdc9bf89d8fb39e" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 14:00:02.157825 6659 raft_consensus.cc:3037] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:00:02.158237 6858 leader_election.cc:334] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:02.158422 6374 catalog_manager.cc:4830] TS 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119): tablet b6cb5119a6db403ba3cae9fa9b121211 (table rebalance_test_table_19 [id=aaf095359a4a40ecbae916f4245a8f5f]) successfully deleted
I20240503 14:00:02.158792 6659 raft_consensus.cc:2454] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 2.
I20240503 14:00:02.159086 6858 leader_election.cc:302] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6
I20240503 14:00:02.159534 9231 tablet_copy_service.cc:342] P ab1d21935d6049888eb0ceccc3c9d735: Request end of tablet copy session 06ca8dbc004c42b99faa527a4e6f9545-9ff9b89e4e3247c59ae48ecfeb72f56b received from {username='slave'} at 127.0.99.133:51367
I20240503 14:00:02.159626 9231 tablet_copy_service.cc:433] P ab1d21935d6049888eb0ceccc3c9d735: ending tablet copy session 06ca8dbc004c42b99faa527a4e6f9545-9ff9b89e4e3247c59ae48ecfeb72f56b on tablet 9ff9b89e4e3247c59ae48ecfeb72f56b with peer 06ca8dbc004c42b99faa527a4e6f9545
I20240503 14:00:02.159663 12748 raft_consensus.cc:2781] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:02.159744 12748 raft_consensus.cc:686] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [term 2 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 14:00:02.159866 12748 consensus_queue.cc:237] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 14:00:02.160122 14177 leader_election.cc:288] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 14:00:02.160996 12063 raft_consensus.cc:482] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.161091 12063 raft_consensus.cc:504] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 14:00:02.161448 12063 leader_election.cc:288] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 14:00:02.161779 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e5875904e6047eb82f28c3cbc9318fc" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 14:00:02.161893 6920 raft_consensus.cc:2454] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 1.
I20240503 14:00:02.162149 9147 leader_election.cc:302] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab1d21935d6049888eb0ceccc3c9d735, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 14:00:02.162303 12063 raft_consensus.cc:2781] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:02.162353 12063 raft_consensus.cc:482] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.162400 12063 raft_consensus.cc:3037] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.162734 6379 catalog_manager.cc:5484] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 1 to 2, leader changed from 0d77752181e944698e809e325e5937e6 (127.0.99.136) to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:02.163086 12063 raft_consensus.cc:504] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 14:00:02.163458 12063 leader_election.cc:288] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 14:00:02.163913 12748 raft_consensus.cc:482] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.163995 12748 raft_consensus.cc:504] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 14:00:02.164171 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e5875904e6047eb82f28c3cbc9318fc" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 14:00:02.164274 6920 raft_consensus.cc:3037] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.164297 12748 leader_election.cc:288] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
W20240503 14:00:02.165093 9148 leader_election.cc:334] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:02.165207 9148 leader_election.cc:334] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:02.165688 6858 leader_election.cc:334] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:02.165891 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e5875904e6047eb82f28c3cbc9318fc" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 14:00:02.166007 9205 raft_consensus.cc:2379] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f58065db70b64ef8b1dc499aa6ae4460 in current term 2: Already voted for candidate ab1d21935d6049888eb0ceccc3c9d735 in this term.
I20240503 14:00:02.166240 6857 leader_election.cc:302] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, ab1d21935d6049888eb0ceccc3c9d735
I20240503 14:00:02.166788 6920 raft_consensus.cc:2454] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1d21935d6049888eb0ceccc3c9d735 in term 2.
I20240503 14:00:02.166893 12748 raft_consensus.cc:2726] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:02.167070 9147 leader_election.cc:302] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab1d21935d6049888eb0ceccc3c9d735, f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6
I20240503 14:00:02.167227 12063 raft_consensus.cc:2781] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:02.167349 12063 raft_consensus.cc:686] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [term 2 LEADER]: Becoming Leader. State: Replica: ab1d21935d6049888eb0ceccc3c9d735, State: Running, Role: LEADER
I20240503 14:00:02.167475 12063 consensus_queue.cc:237] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 14:00:02.169795 6379 catalog_manager.cc:5484] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 reported cstate change: term changed from 1 to 2, leader changed from 0d77752181e944698e809e325e5937e6 (127.0.99.136) to ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "ab1d21935d6049888eb0ceccc3c9d735" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:02.184108 12748 raft_consensus.cc:482] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.184216 12748 raft_consensus.cc:504] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 14:00:02.184659 12748 leader_election.cc:288] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
W20240503 14:00:02.186103 6856 leader_election.cc:334] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:02.186583 6858 leader_election.cc:334] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:02.186646 6858 leader_election.cc:302] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:02.186795 15111 raft_consensus.cc:2726] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:02.206598 15927 raft_consensus.cc:482] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.206745 15927 raft_consensus.cc:504] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 14:00:02.207147 15927 leader_election.cc:288] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 14:00:02.207551 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ed55b431847f39d832ebe61e6c5e4" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 14:00:02.207675 9081 raft_consensus.cc:2454] T 292ed55b431847f39d832ebe61e6c5e4 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 14:00:02.207964 6857 leader_election.cc:302] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, f58065db70b64ef8b1dc499aa6ae4460; no voters:
W20240503 14:00:02.208210 6858 leader_election.cc:334] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:02.208230 15927 raft_consensus.cc:2781] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:02.208303 15927 raft_consensus.cc:482] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:02.208348 15927 raft_consensus.cc:3037] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.209399 15927 raft_consensus.cc:504] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 14:00:02.209776 15927 leader_election.cc:288] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 election: Requested vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 14:00:02.210175 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ed55b431847f39d832ebe61e6c5e4" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 14:00:02.210275 9080 raft_consensus.cc:3037] T 292ed55b431847f39d832ebe61e6c5e4 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.211261 9080 raft_consensus.cc:2454] T 292ed55b431847f39d832ebe61e6c5e4 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 2.
I20240503 14:00:02.211647 6857 leader_election.cc:302] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 14:00:02.211827 15927 raft_consensus.cc:2781] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:02.211984 15927 raft_consensus.cc:686] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 14:00:02.212128 15927 consensus_queue.cc:237] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
W20240503 14:00:02.212915 6858 leader_election.cc:334] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:02.214762 6380 catalog_manager.cc:5484] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 1 to 2, leader changed from 0d77752181e944698e809e325e5937e6 (127.0.99.136) to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: UNKNOWN } } }
W20240503 14:00:02.330825 6596 consensus_peers.cc:485] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:02.331472 6596 consensus_peers.cc:485] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:02.332580 9080 raft_consensus.cc:1261] T 2ddf3f4cf2884b8785b216a59a5b40f9 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:00:02.332999 12063 consensus_queue.cc:1035] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:00:02.336010 9148 consensus_peers.cc:485] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:02.369876 9080 raft_consensus.cc:1261] T e76ded33d780455d963c51a253cbd0e0 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:00:02.370322 14177 consensus_queue.cc:1035] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:00:02.403416 16574 raft_consensus.cc:482] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.403555 16574 raft_consensus.cc:504] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 14:00:02.404018 16574 leader_election.cc:288] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 14:00:02.404342 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b2d8a9ed5f4b09812a85445a8e3b85" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
W20240503 14:00:02.405155 6463 leader_election.cc:334] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.405220 6463 leader_election.cc:302] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da; no voters: 06ca8dbc004c42b99faa527a4e6f9545, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:02.405376 16574 raft_consensus.cc:2726] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:02.437335 9080 raft_consensus.cc:1261] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:00:02.437733 15959 consensus_queue.cc:1035] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:00:02.461406 6858 consensus_peers.cc:485] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:02.469919 8884 consensus_peers.cc:485] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:02.473976 9205 raft_consensus.cc:1261] T da57af787aef40628e2c043023763f02 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:00:02.474408 14390 consensus_queue.cc:1035] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:00:02.474720 8884 consensus_peers.cc:485] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:02.482045 9081 raft_consensus.cc:1261] T 47d38e1d60dd42eabf7191598ded8cd5 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:00:02.482406 16749 consensus_queue.cc:1035] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:00:02.485874 9148 consensus_peers.cc:485] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:02.486698 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:02.490832 9148 consensus_peers.cc:485] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:02.496824 15927 raft_consensus.cc:482] T 95fb9ad43cf34a78b98965035cd086ec P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.496930 15927 raft_consensus.cc:504] T 95fb9ad43cf34a78b98965035cd086ec P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 14:00:02.497262 15927 leader_election.cc:288] T 95fb9ad43cf34a78b98965035cd086ec P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 14:00:02.497627 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fb9ad43cf34a78b98965035cd086ec" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
W20240503 14:00:02.498302 6856 leader_election.cc:334] T 95fb9ad43cf34a78b98965035cd086ec P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.498396 6856 leader_election.cc:302] T 95fb9ad43cf34a78b98965035cd086ec P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 2145d15ab2d74c0287839266f1411a43, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:02.498541 15927 raft_consensus.cc:2726] T 95fb9ad43cf34a78b98965035cd086ec P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:02.504863 6659 raft_consensus.cc:1261] T 6ee85f4483004c1f9499203c7323706d P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:00:02.505496 15927 consensus_queue.cc:1035] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:02.505697 6526 raft_consensus.cc:1261] T e92eaea6e9c8493690b749f162949ac4 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:00:02.506110 14390 consensus_queue.cc:1035] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:00:02.511520 8884 consensus_peers.cc:485] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:02.517470 16749 raft_consensus.cc:482] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.517586 16749 raft_consensus.cc:504] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 14:00:02.517938 16749 leader_election.cc:288] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 14:00:02.518319 6526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387d78763d574e57b914e2dbac83a932" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
W20240503 14:00:02.518920 9146 leader_election.cc:334] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.518985 9146 leader_election.cc:302] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab1d21935d6049888eb0ceccc3c9d735; no voters: 4a85e492c8154e9881a9cc2a105d7e64, 5f46e19644fd4f4b825c86ba6af768da
I20240503 14:00:02.519102 16749 raft_consensus.cc:2726] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:00:02.533814 9081 raft_consensus.cc:1261] T 0b397d0460c94438a773be184312b60f P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:00:02.534262 15927 consensus_queue.cc:1035] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:02.535317 8947 raft_consensus.cc:1261] T 0fe53d65ab4d482aad118d6e3bd5533c P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:00:02.535755 16749 consensus_queue.cc:1035] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:02.540735 8946 raft_consensus.cc:1203] T 9ff9b89e4e3247c59ae48ecfeb72f56b P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240503 14:00:02.543215 6858 consensus_peers.cc:485] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:02.545171 16574 raft_consensus.cc:482] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.545289 16574 raft_consensus.cc:504] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 14:00:02.545594 14486 raft_consensus.cc:482] T b5780f4a85b146e2bfbbf7d2feaca316 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.545688 14486 raft_consensus.cc:504] T b5780f4a85b146e2bfbbf7d2feaca316 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 14:00:02.545732 16574 leader_election.cc:288] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 14:00:02.546036 14486 leader_election.cc:288] T b5780f4a85b146e2bfbbf7d2feaca316 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 14:00:02.546054 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387d78763d574e57b914e2dbac83a932" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 14:00:02.546198 9205 raft_consensus.cc:2454] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 2.
I20240503 14:00:02.546677 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5780f4a85b146e2bfbbf7d2feaca316" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
W20240503 14:00:02.547325 6463 leader_election.cc:334] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.547675 6464 leader_election.cc:302] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, ab1d21935d6049888eb0ceccc3c9d735; no voters: 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:02.547917 16574 raft_consensus.cc:2781] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 14:00:02.547974 16574 raft_consensus.cc:482] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.548019 16574 raft_consensus.cc:3037] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Advancing to term 3
W20240503 14:00:02.548275 9146 leader_election.cc:334] T b5780f4a85b146e2bfbbf7d2feaca316 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.548341 9146 leader_election.cc:302] T b5780f4a85b146e2bfbbf7d2feaca316 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab1d21935d6049888eb0ceccc3c9d735; no voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460
I20240503 14:00:02.548452 14486 raft_consensus.cc:2726] T b5780f4a85b146e2bfbbf7d2feaca316 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:02.549072 16574 raft_consensus.cc:504] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 14:00:02.549402 16574 leader_election.cc:288] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 3 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 14:00:02.549881 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387d78763d574e57b914e2dbac83a932" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 14:00:02.550029 9205 raft_consensus.cc:3037] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:00:02.550998 9205 raft_consensus.cc:2454] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 3.
I20240503 14:00:02.551338 6464 leader_election.cc:302] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 14:00:02.551527 16574 raft_consensus.cc:2781] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:00:02.551616 16574 raft_consensus.cc:686] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da [term 3 LEADER]: Becoming Leader. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Running, Role: LEADER
I20240503 14:00:02.551764 16574 consensus_queue.cc:237] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
W20240503 14:00:02.551920 6463 leader_election.cc:334] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.552168 6659 raft_consensus.cc:1261] T 442c955ab10f4ecbafdc9bf89d8fb39e P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:00:02.552610 13370 consensus_queue.cc:1035] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:02.553354 13370 raft_consensus.cc:482] T 586369c0c3174070a3e184af38b3adf7 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.553452 13370 raft_consensus.cc:504] T 586369c0c3174070a3e184af38b3adf7 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 14:00:02.553800 13370 leader_election.cc:288] T 586369c0c3174070a3e184af38b3adf7 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 14:00:02.554185 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "586369c0c3174070a3e184af38b3adf7" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 14:00:02.554571 6380 catalog_manager.cc:5484] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: term changed from 2 to 3, leader changed from 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) to 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129). New cstate: current_term: 3 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 14:00:02.555990 6856 leader_election.cc:334] T 586369c0c3174070a3e184af38b3adf7 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.556066 6856 leader_election.cc:302] T 586369c0c3174070a3e184af38b3adf7 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 2145d15ab2d74c0287839266f1411a43, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:02.556421 16564 raft_consensus.cc:482] T 957b89de328d434d8ea03a59bc93fdcb P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.556535 16564 raft_consensus.cc:504] T 957b89de328d434d8ea03a59bc93fdcb P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 14:00:02.556715 13370 raft_consensus.cc:2726] T 586369c0c3174070a3e184af38b3adf7 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:02.556910 16564 leader_election.cc:288] T 957b89de328d434d8ea03a59bc93fdcb P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 14:00:02.557288 6526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "957b89de328d434d8ea03a59bc93fdcb" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
W20240503 14:00:02.558048 8882 leader_election.cc:334] T 957b89de328d434d8ea03a59bc93fdcb P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.558112 8882 leader_election.cc:302] T 957b89de328d434d8ea03a59bc93fdcb P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545; no voters: 4a85e492c8154e9881a9cc2a105d7e64, 5f46e19644fd4f4b825c86ba6af768da
I20240503 14:00:02.558337 16564 raft_consensus.cc:2726] T 957b89de328d434d8ea03a59bc93fdcb P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:02.559232 6922 raft_consensus.cc:1261] T a914c22ceff44fc685aad0fdcd04d844 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:00:02.559602 16096 consensus_queue.cc:1035] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:00:02.559780 6858 consensus_peers.cc:485] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:02.562932 9148 consensus_peers.cc:485] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:02.567335 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:02.573143 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:02.577338 6465 consensus_peers.cc:485] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:02.577899 14177 raft_consensus.cc:482] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.577998 14177 raft_consensus.cc:504] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 14:00:02.578399 14177 leader_election.cc:288] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 14:00:02.578888 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ce36eb7c25413eaae380f32b465df7" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
W20240503 14:00:02.579675 6594 leader_election.cc:334] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.579749 6594 leader_election.cc:302] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43; no voters: 4a85e492c8154e9881a9cc2a105d7e64, 8d574fb4a89d4eea947c859470d44dd4
I20240503 14:00:02.579885 14177 raft_consensus.cc:2726] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:00:02.600800 6526 raft_consensus.cc:1261] T 0fa4727297da40eb9f58f5daa8150e0e P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:00:02.601202 14177 consensus_queue.cc:1035] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:02.601665 6526 raft_consensus.cc:1261] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:00:02.602032 16564 consensus_queue.cc:1035] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:00:02.602185 14177 raft_consensus.cc:482] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.602273 14177 raft_consensus.cc:504] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 14:00:02.602635 14177 leader_election.cc:288] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
W20240503 14:00:02.604269 6596 leader_election.cc:334] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:02.606420 6596 consensus_peers.cc:485] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:02.608733 6594 leader_election.cc:334] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.608806 6594 leader_election.cc:302] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:02.608994 15959 raft_consensus.cc:2726] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:02.626571 15959 raft_consensus.cc:482] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.626709 15959 raft_consensus.cc:504] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 14:00:02.627172 15959 leader_election.cc:288] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 14:00:02.627499 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "586369c0c3174070a3e184af38b3adf7" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 14:00:02.627648 6922 raft_consensus.cc:2454] T 586369c0c3174070a3e184af38b3adf7 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 14:00:02.627919 6595 leader_election.cc:302] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 14:00:02.628072 15959 raft_consensus.cc:2781] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:02.628142 15959 raft_consensus.cc:482] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.628193 15959 raft_consensus.cc:3037] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:00:02.628578 6594 leader_election.cc:334] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.629321 15959 raft_consensus.cc:504] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 14:00:02.629629 15959 leader_election.cc:288] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 14:00:02.629917 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "586369c0c3174070a3e184af38b3adf7" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 14:00:02.630034 6922 raft_consensus.cc:3037] T 586369c0c3174070a3e184af38b3adf7 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:00:02.630741 6594 leader_election.cc:334] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.631114 6922 raft_consensus.cc:2454] T 586369c0c3174070a3e184af38b3adf7 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 2.
I20240503 14:00:02.631377 6595 leader_election.cc:302] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters: 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:02.631728 15959 raft_consensus.cc:2781] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:02.631824 15959 raft_consensus.cc:686] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [term 2 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 14:00:02.631954 15959 consensus_queue.cc:237] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 14:00:02.634718 6387 catalog_manager.cc:5484] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 1 to 2, leader changed from 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:02.638418 13370 raft_consensus.cc:482] T 752e0d08505b4c8cb4b2a44cc00ce858 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.638515 13370 raft_consensus.cc:504] T 752e0d08505b4c8cb4b2a44cc00ce858 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 14:00:02.638872 13370 leader_election.cc:288] T 752e0d08505b4c8cb4b2a44cc00ce858 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 14:00:02.639161 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752e0d08505b4c8cb4b2a44cc00ce858" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
W20240503 14:00:02.639981 6856 leader_election.cc:334] T 752e0d08505b4c8cb4b2a44cc00ce858 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.640055 6856 leader_election.cc:302] T 752e0d08505b4c8cb4b2a44cc00ce858 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 2145d15ab2d74c0287839266f1411a43, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:02.640177 13370 raft_consensus.cc:2726] T 752e0d08505b4c8cb4b2a44cc00ce858 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:02.644750 6858 consensus_peers.cc:485] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:02.650125 16574 raft_consensus.cc:482] T 2f3ac8b5d69745d788bb21a316e0ef21 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.650228 16574 raft_consensus.cc:504] T 2f3ac8b5d69745d788bb21a316e0ef21 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 14:00:02.650575 16574 leader_election.cc:288] T 2f3ac8b5d69745d788bb21a316e0ef21 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 14:00:02.650915 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3ac8b5d69745d788bb21a316e0ef21" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
W20240503 14:00:02.651582 6463 leader_election.cc:334] T 2f3ac8b5d69745d788bb21a316e0ef21 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.651659 6463 leader_election.cc:302] T 2f3ac8b5d69745d788bb21a316e0ef21 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da; no voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460
I20240503 14:00:02.651785 16574 raft_consensus.cc:2726] T 2f3ac8b5d69745d788bb21a316e0ef21 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:02.658990 6659 raft_consensus.cc:1261] T 1eec99c0bb994dd7b6ae63918125bccb P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:00:02.659355 12063 consensus_queue.cc:1035] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:00:02.662483 9148 consensus_peers.cc:485] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:02.666417 16009 raft_consensus.cc:482] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.666517 16009 raft_consensus.cc:504] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 14:00:02.666961 16009 leader_election.cc:288] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 14:00:02.667213 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ce36eb7c25413eaae380f32b465df7" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43" is_pre_election: true
I20240503 14:00:02.667344 6659 raft_consensus.cc:2454] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 2.
I20240503 14:00:02.667577 9017 leader_election.cc:302] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 14:00:02.668010 16009 raft_consensus.cc:2781] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 14:00:02.668066 16009 raft_consensus.cc:482] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
W20240503 14:00:02.668110 9015 leader_election.cc:334] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.668232 16009 raft_consensus.cc:3037] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:00:02.669600 16009 raft_consensus.cc:504] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 14:00:02.669940 16009 leader_election.cc:288] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 3 election: Requested vote from peers 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 14:00:02.670189 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ce36eb7c25413eaae380f32b465df7" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2145d15ab2d74c0287839266f1411a43"
I20240503 14:00:02.670291 6659 raft_consensus.cc:3037] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Advancing to term 3
W20240503 14:00:02.671017 9015 leader_election.cc:334] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.671334 6659 raft_consensus.cc:2454] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 3.
I20240503 14:00:02.671622 9017 leader_election.cc:302] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, 8d574fb4a89d4eea947c859470d44dd4; no voters: 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:02.672175 16009 raft_consensus.cc:2781] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:00:02.672263 16009 raft_consensus.cc:686] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 [term 3 LEADER]: Becoming Leader. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 14:00:02.672382 16009 consensus_queue.cc:237] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 14:00:02.675613 6387 catalog_manager.cc:5484] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: term changed from 2 to 3, leader changed from 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) to 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134). New cstate: current_term: 3 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:02.685824 16574 raft_consensus.cc:482] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.685989 16574 raft_consensus.cc:504] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 14:00:02.686414 16574 leader_election.cc:288] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 14:00:02.686738 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea438ff21d9744f697878406493f0fdc" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
W20240503 14:00:02.686983 9148 consensus_peers.cc:485] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:02.687613 6463 leader_election.cc:334] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.687691 6463 leader_election.cc:302] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da; no voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460
I20240503 14:00:02.687817 16574 raft_consensus.cc:2726] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:02.708823 9205 raft_consensus.cc:1261] T abe859639139483aa1e57ee6c9722b87 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:00:02.709283 15099 consensus_queue.cc:1035] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:00:02.725471 12367 raft_consensus.cc:482] T 0a719fd125e54a0fa1c9b42a92548da0 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.725590 12367 raft_consensus.cc:504] T 0a719fd125e54a0fa1c9b42a92548da0 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 14:00:02.725994 12367 leader_election.cc:288] T 0a719fd125e54a0fa1c9b42a92548da0 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 14:00:02.726266 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a719fd125e54a0fa1c9b42a92548da0" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
W20240503 14:00:02.727303 8882 leader_election.cc:334] T 0a719fd125e54a0fa1c9b42a92548da0 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.727396 8882 leader_election.cc:302] T 0a719fd125e54a0fa1c9b42a92548da0 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545; no voters: 4a85e492c8154e9881a9cc2a105d7e64, 8d574fb4a89d4eea947c859470d44dd4
I20240503 14:00:02.727541 12367 raft_consensus.cc:2726] T 0a719fd125e54a0fa1c9b42a92548da0 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:02.732923 14390 raft_consensus.cc:482] T 6321c633e38e4c9e853705b2b271326f P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.733042 14390 raft_consensus.cc:504] T 6321c633e38e4c9e853705b2b271326f P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 14:00:02.733413 14390 leader_election.cc:288] T 6321c633e38e4c9e853705b2b271326f P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 14:00:02.734064 6922 raft_consensus.cc:1261] T 1e5875904e6047eb82f28c3cbc9318fc P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Refusing update from remote peer ab1d21935d6049888eb0ceccc3c9d735: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20240503 14:00:02.734457 8882 leader_election.cc:334] T 6321c633e38e4c9e853705b2b271326f P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.734470 12063 consensus_queue.cc:1035] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:02.734848 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6321c633e38e4c9e853705b2b271326f" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 14:00:02.735226 8883 leader_election.cc:302] T 6321c633e38e4c9e853705b2b271326f P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545; no voters: 4a85e492c8154e9881a9cc2a105d7e64, 8d574fb4a89d4eea947c859470d44dd4
I20240503 14:00:02.735455 14390 raft_consensus.cc:2726] T 6321c633e38e4c9e853705b2b271326f P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:02.777024 12748 raft_consensus.cc:482] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.777135 12748 raft_consensus.cc:504] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 14:00:02.777523 12748 leader_election.cc:288] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 14:00:02.777849 6526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea438ff21d9744f697878406493f0fdc" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 14:00:02.777995 6526 raft_consensus.cc:2454] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 14:00:02.778318 6857 leader_election.cc:302] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
W20240503 14:00:02.778425 6856 leader_election.cc:334] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.778477 12748 raft_consensus.cc:2781] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:02.778517 12748 raft_consensus.cc:482] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.778543 12748 raft_consensus.cc:3037] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.779448 12748 raft_consensus.cc:504] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 14:00:02.779819 12748 leader_election.cc:288] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 14:00:02.780169 6526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea438ff21d9744f697878406493f0fdc" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 14:00:02.780289 6526 raft_consensus.cc:3037] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:00:02.780795 6856 leader_election.cc:334] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.781271 6526 raft_consensus.cc:2454] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 2.
I20240503 14:00:02.781531 6857 leader_election.cc:302] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters: 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:02.781673 12748 raft_consensus.cc:2781] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:02.781759 12748 raft_consensus.cc:686] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 [term 2 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 14:00:02.781912 12748 consensus_queue.cc:237] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 14:00:02.784036 6387 catalog_manager.cc:5484] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 1 to 2, leader changed from 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:02.785346 12748 raft_consensus.cc:482] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.785434 12748 raft_consensus.cc:504] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 14:00:02.785784 12748 leader_election.cc:288] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 14:00:02.786407 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5780f4a85b146e2bfbbf7d2feaca316" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 14:00:02.786556 9205 raft_consensus.cc:2454] T b5780f4a85b146e2bfbbf7d2feaca316 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
W20240503 14:00:02.786777 6856 leader_election.cc:334] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.786872 6857 leader_election.cc:302] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab1d21935d6049888eb0ceccc3c9d735, f58065db70b64ef8b1dc499aa6ae4460; no voters: 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:02.787016 12748 raft_consensus.cc:2781] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:02.787072 12748 raft_consensus.cc:482] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.787111 12748 raft_consensus.cc:3037] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.787885 12748 raft_consensus.cc:504] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 14:00:02.788187 12748 leader_election.cc:288] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 14:00:02.788522 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5780f4a85b146e2bfbbf7d2feaca316" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 14:00:02.788637 9205 raft_consensus.cc:3037] T b5780f4a85b146e2bfbbf7d2feaca316 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:00:02.789163 6856 leader_election.cc:334] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.789947 9205 raft_consensus.cc:2454] T b5780f4a85b146e2bfbbf7d2feaca316 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 2.
I20240503 14:00:02.790431 6857 leader_election.cc:302] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab1d21935d6049888eb0ceccc3c9d735, f58065db70b64ef8b1dc499aa6ae4460; no voters: 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:02.790581 13370 raft_consensus.cc:2781] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:02.790663 13370 raft_consensus.cc:686] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 14:00:02.790796 13370 consensus_queue.cc:237] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 14:00:02.792912 6387 catalog_manager.cc:5484] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 1 to 2, leader changed from 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:02.794237 9081 raft_consensus.cc:1261] T 292ed55b431847f39d832ebe61e6c5e4 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:00:02.794637 13370 consensus_queue.cc:1035] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:00:02.820595 16610 raft_consensus.cc:482] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.820725 16610 raft_consensus.cc:504] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 14:00:02.821084 16610 leader_election.cc:288] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 14:00:02.821511 8946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6321c633e38e4c9e853705b2b271326f" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 14:00:02.821653 8946 raft_consensus.cc:2454] T 6321c633e38e4c9e853705b2b271326f P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 14:00:02.821962 9016 leader_election.cc:302] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters:
W20240503 14:00:02.822160 9015 leader_election.cc:334] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.822175 16610 raft_consensus.cc:2781] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:02.822274 16610 raft_consensus.cc:482] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.822329 16610 raft_consensus.cc:3037] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.823137 15099 raft_consensus.cc:482] T d55d270456d3460586ff3cd41787b5d2 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.823166 16610 raft_consensus.cc:504] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 14:00:02.823268 15099 raft_consensus.cc:504] T d55d270456d3460586ff3cd41787b5d2 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 14:00:02.823568 16610 leader_election.cc:288] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 election: Requested vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 14:00:02.823696 15099 leader_election.cc:288] T d55d270456d3460586ff3cd41787b5d2 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 14:00:02.823963 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55d270456d3460586ff3cd41787b5d2" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 14:00:02.824072 8946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6321c633e38e4c9e853705b2b271326f" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 14:00:02.824167 8946 raft_consensus.cc:3037] T 6321c633e38e4c9e853705b2b271326f P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.825021 8946 raft_consensus.cc:2454] T 6321c633e38e4c9e853705b2b271326f P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 2.
W20240503 14:00:02.825225 6463 leader_election.cc:334] T d55d270456d3460586ff3cd41787b5d2 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:02.825271 9015 leader_election.cc:334] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.825297 6463 leader_election.cc:302] T d55d270456d3460586ff3cd41787b5d2 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da; no voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460
I20240503 14:00:02.825439 15099 raft_consensus.cc:2726] T d55d270456d3460586ff3cd41787b5d2 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:02.825424 9016 leader_election.cc:302] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters: 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:02.825589 16610 raft_consensus.cc:2781] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:02.825665 16610 raft_consensus.cc:686] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 [term 2 LEADER]: Becoming Leader. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 14:00:02.825814 16610 consensus_queue.cc:237] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } }
I20240503 14:00:02.828629 6380 catalog_manager.cc:5484] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: term changed from 1 to 2, leader changed from 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) to 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134). New cstate: current_term: 2 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } }
I20240503 14:00:02.831027 12063 raft_consensus.cc:482] T b08947e5ddd04e7e8dddfa59ca65f713 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.831110 12063 raft_consensus.cc:504] T b08947e5ddd04e7e8dddfa59ca65f713 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 14:00:02.831439 12063 leader_election.cc:288] T b08947e5ddd04e7e8dddfa59ca65f713 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 14:00:02.831751 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b08947e5ddd04e7e8dddfa59ca65f713" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
W20240503 14:00:02.832399 9146 leader_election.cc:334] T b08947e5ddd04e7e8dddfa59ca65f713 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.832469 9146 leader_election.cc:302] T b08947e5ddd04e7e8dddfa59ca65f713 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab1d21935d6049888eb0ceccc3c9d735; no voters: 4a85e492c8154e9881a9cc2a105d7e64, f58065db70b64ef8b1dc499aa6ae4460
I20240503 14:00:02.832592 12063 raft_consensus.cc:2726] T b08947e5ddd04e7e8dddfa59ca65f713 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:02.833179 13370 raft_consensus.cc:482] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.833294 13370 raft_consensus.cc:504] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 14:00:02.833639 13370 leader_election.cc:288] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 14:00:02.834002 6526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3ac8b5d69745d788bb21a316e0ef21" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 14:00:02.834124 6526 raft_consensus.cc:2454] T 2f3ac8b5d69745d788bb21a316e0ef21 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 14:00:02.834379 6857 leader_election.cc:302] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
W20240503 14:00:02.834574 6856 leader_election.cc:334] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.834648 13370 raft_consensus.cc:2781] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:02.834750 13370 raft_consensus.cc:482] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.834926 13370 raft_consensus.cc:3037] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.836139 13370 raft_consensus.cc:504] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 14:00:02.836647 13370 leader_election.cc:288] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 14:00:02.836789 6526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3ac8b5d69745d788bb21a316e0ef21" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 14:00:02.836889 6526 raft_consensus.cc:3037] T 2f3ac8b5d69745d788bb21a316e0ef21 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:00:02.837522 6856 leader_election.cc:334] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.837730 6526 raft_consensus.cc:2454] T 2f3ac8b5d69745d788bb21a316e0ef21 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 2.
I20240503 14:00:02.837980 6857 leader_election.cc:302] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters: 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:02.838164 13370 raft_consensus.cc:2781] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:02.838276 13370 raft_consensus.cc:686] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 14:00:02.838413 13370 consensus_queue.cc:237] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 14:00:02.841078 6380 catalog_manager.cc:5484] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 1 to 2, leader changed from 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:02.896057 16009 raft_consensus.cc:482] T 01551845458048e48df9901947b4eec6 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.896165 16009 raft_consensus.cc:504] T 01551845458048e48df9901947b4eec6 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 14:00:02.896531 16009 leader_election.cc:288] T 01551845458048e48df9901947b4eec6 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547)
I20240503 14:00:02.896855 8946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01551845458048e48df9901947b4eec6" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
W20240503 14:00:02.897372 9015 leader_election.cc:334] T 01551845458048e48df9901947b4eec6 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.897437 9015 leader_election.cc:302] T 01551845458048e48df9901947b4eec6 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d574fb4a89d4eea947c859470d44dd4; no voters: 06ca8dbc004c42b99faa527a4e6f9545, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:02.897549 16009 raft_consensus.cc:2726] T 01551845458048e48df9901947b4eec6 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:02.905354 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:02.921777 6372 catalog_manager.cc:4573] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3856ff74b3bf4f608e3c6cdd06ce144e: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.923586 15099 raft_consensus.cc:482] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.923699 15099 raft_consensus.cc:504] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 14:00:02.924105 15099 leader_election.cc:288] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
W20240503 14:00:02.925364 6463 leader_election.cc:334] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.932615 14177 raft_consensus.cc:482] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.932713 14177 raft_consensus.cc:504] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 14:00:02.933053 14177 leader_election.cc:288] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 14:00:02.933255 12063 raft_consensus.cc:482] T ce9410a4ed09491b8d37456ae44880a1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.933387 12063 raft_consensus.cc:504] T ce9410a4ed09491b8d37456ae44880a1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 14:00:02.933534 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752e0d08505b4c8cb4b2a44cc00ce858" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 14:00:02.933655 6922 raft_consensus.cc:2454] T 752e0d08505b4c8cb4b2a44cc00ce858 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 14:00:02.933693 8946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "957b89de328d434d8ea03a59bc93fdcb" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 14:00:02.933771 12063 leader_election.cc:288] T ce9410a4ed09491b8d37456ae44880a1 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 14:00:02.933812 8946 raft_consensus.cc:2454] T 957b89de328d434d8ea03a59bc93fdcb P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 14:00:02.934057 6464 leader_election.cc:302] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 5f46e19644fd4f4b825c86ba6af768da; no voters: 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:02.934070 6595 leader_election.cc:302] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 14:00:02.934232 14177 raft_consensus.cc:2781] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:02.934283 14177 raft_consensus.cc:482] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.934319 14177 raft_consensus.cc:3037] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.934422 15099 raft_consensus.cc:2781] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:02.934479 15099 raft_consensus.cc:482] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.934451 6526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce9410a4ed09491b8d37456ae44880a1" candidate_uuid: "ab1d21935d6049888eb0ceccc3c9d735" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 14:00:02.934517 15099 raft_consensus.cc:3037] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.935362 15099 raft_consensus.cc:504] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 14:00:02.935356 14177 raft_consensus.cc:504] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 14:00:02.935655 14177 leader_election.cc:288] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531)
I20240503 14:00:02.935657 15099 leader_election.cc:288] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 election: Requested vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 14:00:02.935925 8946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "957b89de328d434d8ea03a59bc93fdcb" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 14:00:02.936036 8946 raft_consensus.cc:3037] T 957b89de328d434d8ea03a59bc93fdcb P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.936412 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752e0d08505b4c8cb4b2a44cc00ce858" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 14:00:02.936527 6922 raft_consensus.cc:3037] T 752e0d08505b4c8cb4b2a44cc00ce858 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.937054 8946 raft_consensus.cc:2454] T 957b89de328d434d8ea03a59bc93fdcb P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 2.
W20240503 14:00:02.937140 6463 leader_election.cc:334] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.937438 6464 leader_election.cc:302] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 5f46e19644fd4f4b825c86ba6af768da; no voters: 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:02.937578 6922 raft_consensus.cc:2454] T 752e0d08505b4c8cb4b2a44cc00ce858 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 2.
I20240503 14:00:02.937619 15099 raft_consensus.cc:2781] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:02.937718 15099 raft_consensus.cc:686] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da [term 2 LEADER]: Becoming Leader. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Running, Role: LEADER
I20240503 14:00:02.937860 6595 leader_election.cc:302] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 14:00:02.937861 15099 consensus_queue.cc:237] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
W20240503 14:00:02.938457 6594 leader_election.cc:334] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.938462 14177 raft_consensus.cc:2781] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:02.938583 14177 raft_consensus.cc:686] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 [term 2 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
W20240503 14:00:02.938604 6594 leader_election.cc:334] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.938696 14177 consensus_queue.cc:237] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 14:00:02.941448 6380 catalog_manager.cc:5484] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: term changed from 1 to 2, leader changed from 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) to 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129). New cstate: current_term: 2 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 14:00:02.943253 9146 leader_election.cc:334] T ce9410a4ed09491b8d37456ae44880a1 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.943365 9146 leader_election.cc:302] T ce9410a4ed09491b8d37456ae44880a1 P ab1d21935d6049888eb0ceccc3c9d735 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab1d21935d6049888eb0ceccc3c9d735; no voters: 4a85e492c8154e9881a9cc2a105d7e64, 5f46e19644fd4f4b825c86ba6af768da
I20240503 14:00:02.943501 12063 raft_consensus.cc:2726] T ce9410a4ed09491b8d37456ae44880a1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:02.943742 6387 catalog_manager.cc:5484] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 1 to 2, leader changed from 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:02.943838 16574 raft_consensus.cc:482] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.943923 16574 raft_consensus.cc:504] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 14:00:02.944232 16574 leader_election.cc:288] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 14:00:02.944700 9205 raft_consensus.cc:1261] T 387d78763d574e57b914e2dbac83a932 P ab1d21935d6049888eb0ceccc3c9d735 [term 3 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240503 14:00:02.944878 9210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce9410a4ed09491b8d37456ae44880a1" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 14:00:02.944995 9210 raft_consensus.cc:2454] T ce9410a4ed09491b8d37456ae44880a1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 1.
I20240503 14:00:02.945416 6464 leader_election.cc:302] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, ab1d21935d6049888eb0ceccc3c9d735; no voters:
I20240503 14:00:02.945590 16798 raft_consensus.cc:2781] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:02.945648 16798 raft_consensus.cc:482] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.945685 16798 raft_consensus.cc:3037] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.945417 16574 consensus_queue.cc:1035] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240503 14:00:02.946816 16798 raft_consensus.cc:504] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
I20240503 14:00:02.947163 12748 raft_consensus.cc:482] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.947252 12748 raft_consensus.cc:504] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 14:00:02.947604 12748 leader_election.cc:288] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 14:00:02.948103 6526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55d270456d3460586ff3cd41787b5d2" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 14:00:02.948237 6526 raft_consensus.cc:2454] T d55d270456d3460586ff3cd41787b5d2 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 14:00:02.948539 6857 leader_election.cc:302] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 14:00:02.948716 12748 raft_consensus.cc:2781] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:02.948773 12748 raft_consensus.cc:482] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.948812 12748 raft_consensus.cc:3037] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.948796 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce9410a4ed09491b8d37456ae44880a1" candidate_uuid: "5f46e19644fd4f4b825c86ba6af768da" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 14:00:02.948917 9205 raft_consensus.cc:3037] T ce9410a4ed09491b8d37456ae44880a1 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:00:02.948971 6856 leader_election.cc:334] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.949795 12748 raft_consensus.cc:504] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 14:00:02.949982 9205 raft_consensus.cc:2454] T ce9410a4ed09491b8d37456ae44880a1 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46e19644fd4f4b825c86ba6af768da in term 2.
I20240503 14:00:02.950093 12748 leader_election.cc:288] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 14:00:02.948139 16798 leader_election.cc:288] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091)
I20240503 14:00:02.950469 6464 leader_election.cc:302] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, ab1d21935d6049888eb0ceccc3c9d735; no voters:
W20240503 14:00:02.950551 6463 leader_election.cc:334] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.950736 15099 raft_consensus.cc:2781] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:02.950820 15099 raft_consensus.cc:686] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [term 2 LEADER]: Becoming Leader. State: Replica: 5f46e19644fd4f4b825c86ba6af768da, State: Running, Role: LEADER
I20240503 14:00:02.950882 6526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55d270456d3460586ff3cd41787b5d2" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 14:00:02.950971 6526 raft_consensus.cc:3037] T d55d270456d3460586ff3cd41787b5d2 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.950945 15099 consensus_queue.cc:237] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } }
W20240503 14:00:02.951599 6856 leader_election.cc:334] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.951902 6526 raft_consensus.cc:2454] T d55d270456d3460586ff3cd41787b5d2 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 2.
I20240503 14:00:02.952584 6857 leader_election.cc:302] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, f58065db70b64ef8b1dc499aa6ae4460; no voters: 4a85e492c8154e9881a9cc2a105d7e64
W20240503 14:00:02.954007 6463 leader_election.cc:334] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.954218 12748 raft_consensus.cc:2781] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:02.954295 12748 raft_consensus.cc:686] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 14:00:02.954411 12748 consensus_queue.cc:237] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } }
I20240503 14:00:02.955090 6387 catalog_manager.cc:5484] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da reported cstate change: term changed from 1 to 2, leader changed from 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) to 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129). New cstate: current_term: 2 leader_uuid: "5f46e19644fd4f4b825c86ba6af768da" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:02.957051 6380 catalog_manager.cc:5484] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 1 to 2, leader changed from 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:00:02.957331 6463 consensus_peers.cc:485] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:02.957505 15959 raft_consensus.cc:482] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.957610 15959 raft_consensus.cc:504] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 14:00:02.957975 15959 leader_election.cc:288] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 14:00:02.958418 14390 raft_consensus.cc:482] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.958515 14390 raft_consensus.cc:504] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 14:00:02.958884 14390 leader_election.cc:288] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
I20240503 14:00:02.959216 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01551845458048e48df9901947b4eec6" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4" is_pre_election: true
I20240503 14:00:02.959386 9081 raft_consensus.cc:2454] T 01551845458048e48df9901947b4eec6 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 14:00:02.959657 8883 leader_election.cc:302] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters:
I20240503 14:00:02.959807 14390 raft_consensus.cc:2781] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:02.959854 14390 raft_consensus.cc:482] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.959892 14390 raft_consensus.cc:3037] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.960901 14390 raft_consensus.cc:504] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 14:00:02.961213 14390 leader_election.cc:288] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134:39481)
W20240503 14:00:02.961724 8882 leader_election.cc:334] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.962074 9081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01551845458048e48df9901947b4eec6" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8d574fb4a89d4eea947c859470d44dd4"
I20240503 14:00:02.962177 9081 raft_consensus.cc:3037] T 01551845458048e48df9901947b4eec6 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.963181 9081 raft_consensus.cc:2454] T 01551845458048e48df9901947b4eec6 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 2.
W20240503 14:00:02.963389 8882 leader_election.cc:334] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.963471 8883 leader_election.cc:302] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters: 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:02.963660 14390 raft_consensus.cc:2781] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:02.963630 6922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fb9ad43cf34a78b98965035cd086ec" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460" is_pre_election: true
I20240503 14:00:02.963732 14390 raft_consensus.cc:686] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 14:00:02.963739 6922 raft_consensus.cc:2454] T 95fb9ad43cf34a78b98965035cd086ec P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 1.
I20240503 14:00:02.963855 14390 consensus_queue.cc:237] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 14:00:02.964006 6595 leader_election.cc:302] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 14:00:02.964279 15959 raft_consensus.cc:2781] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:02.964336 15959 raft_consensus.cc:482] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.964372 15959 raft_consensus.cc:3037] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.965359 15959 raft_consensus.cc:504] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 14:00:02.965669 15959 leader_election.cc:288] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 election: Requested vote from peers f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132:43531), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 14:00:02.966296 6380 catalog_manager.cc:5484] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 1 to 2, leader changed from 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 2 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:02.967319 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fb9ad43cf34a78b98965035cd086ec" candidate_uuid: "2145d15ab2d74c0287839266f1411a43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f58065db70b64ef8b1dc499aa6ae4460"
I20240503 14:00:02.967420 6920 raft_consensus.cc:3037] T 95fb9ad43cf34a78b98965035cd086ec P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.968436 6920 raft_consensus.cc:2454] T 95fb9ad43cf34a78b98965035cd086ec P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2145d15ab2d74c0287839266f1411a43 in term 2.
I20240503 14:00:02.968716 6595 leader_election.cc:302] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 14:00:02.968892 15959 raft_consensus.cc:2781] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:02.968988 15959 raft_consensus.cc:686] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [term 2 LEADER]: Becoming Leader. State: Replica: 2145d15ab2d74c0287839266f1411a43, State: Running, Role: LEADER
I20240503 14:00:02.969125 15959 consensus_queue.cc:237] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
W20240503 14:00:02.969414 6594 leader_election.cc:334] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:02.970081 6594 leader_election.cc:334] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.971673 6380 catalog_manager.cc:5484] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 reported cstate change: term changed from 1 to 2, leader changed from 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) to 2145d15ab2d74c0287839266f1411a43 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:02.974820 13370 raft_consensus.cc:482] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.974912 13370 raft_consensus.cc:504] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 14:00:02.975219 13370 leader_election.cc:288] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 14:00:02.975622 9210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b08947e5ddd04e7e8dddfa59ca65f713" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735" is_pre_election: true
I20240503 14:00:02.975757 9210 raft_consensus.cc:2454] T b08947e5ddd04e7e8dddfa59ca65f713 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 1.
I20240503 14:00:02.976042 6857 leader_election.cc:302] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab1d21935d6049888eb0ceccc3c9d735, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 14:00:02.976181 13370 raft_consensus.cc:2781] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:02.976225 13370 raft_consensus.cc:482] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.976266 13370 raft_consensus.cc:3037] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:00:02.976305 6856 leader_election.cc:334] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.977288 13370 raft_consensus.cc:504] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
I20240503 14:00:02.977596 13370 leader_election.cc:288] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 election: Requested vote from peers ab1d21935d6049888eb0ceccc3c9d735 (127.0.99.135:34091), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 14:00:02.977954 9210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b08947e5ddd04e7e8dddfa59ca65f713" candidate_uuid: "f58065db70b64ef8b1dc499aa6ae4460" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab1d21935d6049888eb0ceccc3c9d735"
I20240503 14:00:02.978094 9210 raft_consensus.cc:3037] T b08947e5ddd04e7e8dddfa59ca65f713 P ab1d21935d6049888eb0ceccc3c9d735 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.978821 9210 raft_consensus.cc:2454] T b08947e5ddd04e7e8dddfa59ca65f713 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58065db70b64ef8b1dc499aa6ae4460 in term 2.
I20240503 14:00:02.979084 6857 leader_election.cc:302] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab1d21935d6049888eb0ceccc3c9d735, f58065db70b64ef8b1dc499aa6ae4460; no voters:
I20240503 14:00:02.979256 13370 raft_consensus.cc:2781] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:02.979369 13370 raft_consensus.cc:686] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 LEADER]: Becoming Leader. State: Replica: f58065db70b64ef8b1dc499aa6ae4460, State: Running, Role: LEADER
I20240503 14:00:02.979496 13370 consensus_queue.cc:237] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } }
W20240503 14:00:02.979615 6856 leader_election.cc:334] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.981823 6380 catalog_manager.cc:5484] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 reported cstate change: term changed from 1 to 2, leader changed from 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) to f58065db70b64ef8b1dc499aa6ae4460 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "f58065db70b64ef8b1dc499aa6ae4460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } health_report { overall_health: HEALTHY } } }
I20240503 14:00:02.984750 16009 raft_consensus.cc:482] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.984834 16009 raft_consensus.cc:504] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 14:00:02.985231 16009 leader_election.cc:288] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 14:00:02.985713 8946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a719fd125e54a0fa1c9b42a92548da0" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545" is_pre_election: true
I20240503 14:00:02.985849 8946 raft_consensus.cc:2454] T 0a719fd125e54a0fa1c9b42a92548da0 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 1.
I20240503 14:00:02.986112 9016 leader_election.cc:302] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters:
W20240503 14:00:02.986197 9015 leader_election.cc:334] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.986263 16009 raft_consensus.cc:2781] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:02.986301 16009 raft_consensus.cc:482] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.986326 16009 raft_consensus.cc:3037] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.987059 16009 raft_consensus.cc:504] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 14:00:02.987380 16009 leader_election.cc:288] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 election: Requested vote from peers 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
I20240503 14:00:02.987902 8946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a719fd125e54a0fa1c9b42a92548da0" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "06ca8dbc004c42b99faa527a4e6f9545"
I20240503 14:00:02.988010 8946 raft_consensus.cc:3037] T 0a719fd125e54a0fa1c9b42a92548da0 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:02.988307 13370 raft_consensus.cc:482] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
W20240503 14:00:02.988418 9015 leader_election.cc:334] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.988394 13370 raft_consensus.cc:504] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 14:00:02.988795 13370 leader_election.cc:288] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 14:00:02.989065 8946 raft_consensus.cc:2454] T 0a719fd125e54a0fa1c9b42a92548da0 P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 2.
I20240503 14:00:02.989652 9016 leader_election.cc:302] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 8d574fb4a89d4eea947c859470d44dd4; no voters: 4a85e492c8154e9881a9cc2a105d7e64
W20240503 14:00:02.989966 6856 leader_election.cc:334] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:02.990154 6858 leader_election.cc:334] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:02.990214 6858 leader_election.cc:302] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:02.990394 16009 raft_consensus.cc:2781] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:02.990473 16009 raft_consensus.cc:686] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 LEADER]: Becoming Leader. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 14:00:02.990602 16009 consensus_queue.cc:237] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } }
I20240503 14:00:02.991168 12367 raft_consensus.cc:482] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.991259 12367 raft_consensus.cc:504] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 14:00:02.991636 12367 leader_election.cc:288] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 14:00:02.992004 6526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b2d8a9ed5f4b09812a85445a8e3b85" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 14:00:02.992076 13370 raft_consensus.cc:2726] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:02.992142 6526 raft_consensus.cc:2454] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 1.
I20240503 14:00:02.992624 8883 leader_election.cc:302] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 5f46e19644fd4f4b825c86ba6af768da; no voters:
I20240503 14:00:02.992767 12367 raft_consensus.cc:2781] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:00:02.992825 12367 raft_consensus.cc:482] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:02.992885 12367 raft_consensus.cc:3037] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:00:02.993007 8882 leader_election.cc:334] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.993814 12367 raft_consensus.cc:504] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 14:00:02.994174 12367 leader_election.cc:288] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 election: Requested vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021)
I20240503 14:00:02.994133 6380 catalog_manager.cc:5484] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: term changed from 1 to 2, leader changed from 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) to 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134). New cstate: current_term: 2 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:02.994508 6526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b2d8a9ed5f4b09812a85445a8e3b85" candidate_uuid: "06ca8dbc004c42b99faa527a4e6f9545" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 14:00:02.994618 6526 raft_consensus.cc:3037] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:00:02.995070 8882 leader_election.cc:334] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:02.995366 6526 raft_consensus.cc:2454] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ca8dbc004c42b99faa527a4e6f9545 in term 2.
I20240503 14:00:02.995648 8883 leader_election.cc:302] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06ca8dbc004c42b99faa527a4e6f9545, 5f46e19644fd4f4b825c86ba6af768da; no voters: 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:02.995865 12367 raft_consensus.cc:2781] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:02.995960 12367 raft_consensus.cc:686] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 LEADER]: Becoming Leader. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Running, Role: LEADER
I20240503 14:00:02.996112 12367 consensus_queue.cc:237] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } }
I20240503 14:00:02.998191 6380 catalog_manager.cc:5484] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 reported cstate change: term changed from 1 to 2, leader changed from 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131) to 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133). New cstate: current_term: 2 leader_uuid: "06ca8dbc004c42b99faa527a4e6f9545" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 14:00:03.027189 6594 consensus_peers.cc:485] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:03.044229 12063 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 14:00:03.054332 6922 raft_consensus.cc:1261] T 586369c0c3174070a3e184af38b3adf7 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:00:03.055192 15959 consensus_queue.cc:1035] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:03.060575 8946 raft_consensus.cc:3037] T 586369c0c3174070a3e184af38b3adf7 P 06ca8dbc004c42b99faa527a4e6f9545 [term 1 LEARNER]: Advancing to term 2
I20240503 14:00:03.061964 8946 raft_consensus.cc:1261] T 586369c0c3174070a3e184af38b3adf7 P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 LEARNER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20240503 14:00:03.064133 6465 consensus_peers.cc:485] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:00:03.064669 15959 consensus_queue.cc:1035] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:03.067185 15959 raft_consensus.cc:1050] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43: attempting to promote NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 to VOTER
I20240503 14:00:03.067572 15959 consensus_queue.cc:237] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
I20240503 14:00:03.069222 8946 raft_consensus.cc:1261] T 586369c0c3174070a3e184af38b3adf7 P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 LEARNER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:03.069809 6922 raft_consensus.cc:1261] T 586369c0c3174070a3e184af38b3adf7 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:03.069797 15959 consensus_queue.cc:1035] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:03.070351 15959 consensus_queue.cc:1035] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 14:00:03.071547 6594 consensus_peers.cc:485] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:03.072407 15876 raft_consensus.cc:2932] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 14:00:03.072837 8946 raft_consensus.cc:2932] T 586369c0c3174070a3e184af38b3adf7 P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 14:00:03.074395 6922 raft_consensus.cc:2932] T 586369c0c3174070a3e184af38b3adf7 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 14:00:03.075906 6380 catalog_manager.cc:5484] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index 4 to 6, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 14:00:03.083600 6858 consensus_peers.cc:485] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:03.117703 6465 consensus_peers.cc:485] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:03.156580 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:03.160370 6526 raft_consensus.cc:1261] T ea438ff21d9744f697878406493f0fdc P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:00:03.160737 13370 consensus_queue.cc:1035] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:00:03.164083 6856 consensus_peers.cc:485] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:03.165182 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:03.179611 8884 consensus_peers.cc:485] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:03.197281 9015 consensus_peers.cc:485] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:03.207824 6465 consensus_peers.cc:485] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:00:03.217216 6658 raft_consensus.cc:1261] T 96ce36eb7c25413eaae380f32b465df7 P 2145d15ab2d74c0287839266f1411a43 [term 3 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240503 14:00:03.218088 16610 consensus_queue.cc:1035] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 14:00:03.242905 9148 consensus_peers.cc:485] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:03.244720 6858 consensus_peers.cc:485] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:03.258770 9017 consensus_peers.cc:485] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:03.267150 9017 consensus_peers.cc:485] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:03.289815 8884 consensus_peers.cc:485] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:03.294178 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:03.298079 9210 raft_consensus.cc:1261] T b5780f4a85b146e2bfbbf7d2feaca316 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:00:03.298508 12748 consensus_queue.cc:1035] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:00:03.301242 9017 consensus_peers.cc:485] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:03.304455 6856 consensus_peers.cc:485] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:03.304930 9017 consensus_peers.cc:485] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:00:03.321260 8946 raft_consensus.cc:1261] T 6321c633e38e4c9e853705b2b271326f P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:00:03.321699 16823 consensus_queue.cc:1035] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:00:03.327014 9015 consensus_peers.cc:485] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:03.327932 6856 consensus_peers.cc:485] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:03.337816 6463 consensus_peers.cc:485] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:03.352200 6463 consensus_peers.cc:485] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:03.370373 6526 raft_consensus.cc:1261] T 2f3ac8b5d69745d788bb21a316e0ef21 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:00:03.370759 15927 consensus_queue.cc:1035] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:00:03.375097 6595 consensus_peers.cc:485] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 -> Peer 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133:34547): Couldn't send request to peer 06ca8dbc004c42b99faa527a4e6f9545. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 95fb9ad43cf34a78b98965035cd086ec. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:03.378301 9015 consensus_peers.cc:485] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:03.382184 6526 raft_consensus.cc:1261] T d55d270456d3460586ff3cd41787b5d2 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:00:03.382548 12748 consensus_queue.cc:1035] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:00:03.385540 6856 consensus_peers.cc:485] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:03.385551 6858 consensus_peers.cc:485] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:03.414775 8946 raft_consensus.cc:1261] T 0a719fd125e54a0fa1c9b42a92548da0 P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:00:03.415355 16009 consensus_queue.cc:1035] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:00:03.419893 16823 raft_consensus.cc:482] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:03.419994 16823 raft_consensus.cc:504] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 14:00:03.421038 16823 leader_election.cc:288] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
W20240503 14:00:03.421689 6594 consensus_peers.cc:485] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:03.421727 9017 leader_election.cc:334] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:03.422058 6526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d9906a118744dd6a78d50e52f1961e2" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da" is_pre_election: true
I20240503 14:00:03.422206 6526 raft_consensus.cc:2454] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 2.
I20240503 14:00:03.422528 9016 leader_election.cc:302] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, 8d574fb4a89d4eea947c859470d44dd4; no voters: 0d77752181e944698e809e325e5937e6
I20240503 14:00:03.422780 16823 raft_consensus.cc:2781] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 14:00:03.422904 16823 raft_consensus.cc:482] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:03.422966 16823 raft_consensus.cc:3037] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:00:03.424079 16823 raft_consensus.cc:504] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
I20240503 14:00:03.424466 16823 leader_election.cc:288] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 3 election: Requested vote from peers 5f46e19644fd4f4b825c86ba6af768da (127.0.99.129:38021), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
I20240503 14:00:03.424763 6526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d9906a118744dd6a78d50e52f1961e2" candidate_uuid: "8d574fb4a89d4eea947c859470d44dd4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f46e19644fd4f4b825c86ba6af768da"
I20240503 14:00:03.424890 6526 raft_consensus.cc:3037] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Advancing to term 3
W20240503 14:00:03.425431 9017 leader_election.cc:334] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:03.425966 6526 raft_consensus.cc:2454] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d574fb4a89d4eea947c859470d44dd4 in term 3.
I20240503 14:00:03.426235 9016 leader_election.cc:302] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da, 8d574fb4a89d4eea947c859470d44dd4; no voters: 0d77752181e944698e809e325e5937e6
I20240503 14:00:03.426434 16823 raft_consensus.cc:2781] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:00:03.426577 16823 raft_consensus.cc:686] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [term 3 LEADER]: Becoming Leader. State: Replica: 8d574fb4a89d4eea947c859470d44dd4, State: Running, Role: LEADER
I20240503 14:00:03.426965 16823 consensus_queue.cc:237] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } }
W20240503 14:00:03.427227 6594 consensus_peers.cc:485] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:03.429668 6379 catalog_manager.cc:5484] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 reported cstate change: term changed from 1 to 3, leader changed from 0d77752181e944698e809e325e5937e6 (127.0.99.136) to 8d574fb4a89d4eea947c859470d44dd4 (127.0.99.134). New cstate: current_term: 3 leader_uuid: "8d574fb4a89d4eea947c859470d44dd4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 14:00:03.435213 8882 consensus_peers.cc:485] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:03.441689 6858 consensus_peers.cc:485] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:00:03.449525 6920 raft_consensus.cc:1261] T 95fb9ad43cf34a78b98965035cd086ec P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:00:03.449910 16819 consensus_queue.cc:1035] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:00:03.450258 9017 consensus_peers.cc:485] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:00:03.455058 6920 raft_consensus.cc:1261] T 752e0d08505b4c8cb4b2a44cc00ce858 P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:00:03.455471 16819 consensus_queue.cc:1035] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:00:03.459246 6596 consensus_peers.cc:485] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:03.466547 6526 raft_consensus.cc:1261] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 5f46e19644fd4f4b825c86ba6af768da [term 2 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:00:03.466885 16564 consensus_queue.cc:1035] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:00:03.475812 8882 consensus_peers.cc:485] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:03.496065 9210 raft_consensus.cc:1261] T ce9410a4ed09491b8d37456ae44880a1 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:00:03.496524 16574 consensus_queue.cc:1035] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 14:00:03.497732 9148 consensus_peers.cc:485] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:00:03.517056 9210 raft_consensus.cc:1261] T b08947e5ddd04e7e8dddfa59ca65f713 P ab1d21935d6049888eb0ceccc3c9d735 [term 2 FOLLOWER]: Refusing update from remote peer f58065db70b64ef8b1dc499aa6ae4460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:00:03.517472 16828 consensus_queue.cc:1035] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1d21935d6049888eb0ceccc3c9d735" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:00:03.521236 6856 consensus_peers.cc:485] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:03.532562 8946 raft_consensus.cc:1261] T 957b89de328d434d8ea03a59bc93fdcb P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 FOLLOWER]: Refusing update from remote peer 5f46e19644fd4f4b825c86ba6af768da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:00:03.533061 16574 consensus_queue.cc:1035] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:00:03.537639 9017 consensus_peers.cc:485] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:03.545717 9017 consensus_peers.cc:485] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:03.550246 6594 consensus_peers.cc:485] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:00:03.552477 9081 raft_consensus.cc:1261] T 01551845458048e48df9901947b4eec6 P 8d574fb4a89d4eea947c859470d44dd4 [term 2 FOLLOWER]: Refusing update from remote peer 06ca8dbc004c42b99faa527a4e6f9545: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:00:03.552881 12367 consensus_queue.cc:1035] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d574fb4a89d4eea947c859470d44dd4" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39481 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:00:03.574187 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:03.609669 12748 raft_consensus.cc:482] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:03.609817 12748 raft_consensus.cc:504] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 14:00:03.610291 12748 leader_election.cc:288] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
W20240503 14:00:03.611348 6858 leader_election.cc:334] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:03.611585 6856 leader_election.cc:334] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:03.611685 6856 leader_election.cc:302] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:03.611841 12748 raft_consensus.cc:2726] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:03.614326 12063 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:03.617353 6596 consensus_peers.cc:485] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:03.627780 6858 consensus_peers.cc:485] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:03.690696 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:03.712668 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:03.725602 9017 consensus_peers.cc:485] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:03.758996 9148 consensus_peers.cc:485] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:00:03.759266 16837 raft_consensus.cc:482] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:03.759393 16837 raft_consensus.cc:504] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 14:00:03.759789 16837 leader_election.cc:288] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
W20240503 14:00:03.760617 6463 leader_election.cc:334] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:03.760869 6465 leader_election.cc:334] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:03.760936 6465 leader_election.cc:302] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:03.761060 16837 raft_consensus.cc:2726] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:03.784117 9146 consensus_peers.cc:485] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:03.786662 6596 consensus_peers.cc:485] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:00:03.800555 15959 raft_consensus.cc:482] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:03.800665 15959 raft_consensus.cc:504] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 14:00:03.801036 15959 leader_election.cc:288] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
W20240503 14:00:03.801978 6596 leader_election.cc:334] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:03.802155 6594 leader_election.cc:334] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:03.802213 6594 leader_election.cc:302] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:03.802341 15959 raft_consensus.cc:2726] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:03.809742 9148 consensus_peers.cc:485] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:03.810242 6596 consensus_peers.cc:485] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:03.840951 9015 consensus_peers.cc:485] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:03.843921 8884 consensus_peers.cc:485] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:00:03.851173 16841 ts_tablet_manager.cc:902] T 95fb9ad43cf34a78b98965035cd086ec P 06ca8dbc004c42b99faa527a4e6f9545: Initiating tablet copy from peer 2145d15ab2d74c0287839266f1411a43 (127.0.99.130:46119)
I20240503 14:00:03.851678 16841 tablet_copy_client.cc:308] T 95fb9ad43cf34a78b98965035cd086ec P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46119
I20240503 14:00:03.852121 6680 tablet_copy_service.cc:140] P 2145d15ab2d74c0287839266f1411a43: Received BeginTabletCopySession request for tablet 95fb9ad43cf34a78b98965035cd086ec from peer 06ca8dbc004c42b99faa527a4e6f9545 ({username='slave'} at 127.0.99.133:36361)
I20240503 14:00:03.852227 6680 tablet_copy_service.cc:161] P 2145d15ab2d74c0287839266f1411a43: Beginning new tablet copy session on tablet 95fb9ad43cf34a78b98965035cd086ec from peer 06ca8dbc004c42b99faa527a4e6f9545 at {username='slave'} at 127.0.99.133:36361: session id = 06ca8dbc004c42b99faa527a4e6f9545-95fb9ad43cf34a78b98965035cd086ec
I20240503 14:00:03.853176 6680 tablet_copy_source_session.cc:205] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:03.854156 16841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95fb9ad43cf34a78b98965035cd086ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:03.856566 16841 tablet_copy_client.cc:783] T 95fb9ad43cf34a78b98965035cd086ec P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:03.856717 16841 tablet_copy_client.cc:647] T 95fb9ad43cf34a78b98965035cd086ec P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:03.857725 16841 tablet_copy_client.cc:523] T 95fb9ad43cf34a78b98965035cd086ec P 06ca8dbc004c42b99faa527a4e6f9545: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:03.858809 16841 tablet_bootstrap.cc:492] T 95fb9ad43cf34a78b98965035cd086ec P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap starting.
W20240503 14:00:03.861199 9146 consensus_peers.cc:485] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:03.877075 9017 consensus_peers.cc:485] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:03.878397 16841 tablet_bootstrap.cc:492] T 95fb9ad43cf34a78b98965035cd086ec P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:03.878629 16841 tablet_bootstrap.cc:492] T 95fb9ad43cf34a78b98965035cd086ec P 06ca8dbc004c42b99faa527a4e6f9545: Bootstrap complete.
I20240503 14:00:03.878819 16841 ts_tablet_manager.cc:1367] T 95fb9ad43cf34a78b98965035cd086ec P 06ca8dbc004c42b99faa527a4e6f9545: Time spent bootstrapping tablet: real 0.020s user 0.016s sys 0.001s
I20240503 14:00:03.879367 16841 raft_consensus.cc:348] T 95fb9ad43cf34a78b98965035cd086ec P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 14:00:03.879520 16841 raft_consensus.cc:729] T 95fb9ad43cf34a78b98965035cd086ec P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 06ca8dbc004c42b99faa527a4e6f9545, State: Initialized, Role: LEARNER
I20240503 14:00:03.879627 16841 consensus_queue.cc:260] T 95fb9ad43cf34a78b98965035cd086ec P 06ca8dbc004c42b99faa527a4e6f9545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: true } }
I20240503 14:00:03.879954 16841 ts_tablet_manager.cc:1398] T 95fb9ad43cf34a78b98965035cd086ec P 06ca8dbc004c42b99faa527a4e6f9545: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 14:00:03.880470 6680 tablet_copy_service.cc:342] P 2145d15ab2d74c0287839266f1411a43: Request end of tablet copy session 06ca8dbc004c42b99faa527a4e6f9545-95fb9ad43cf34a78b98965035cd086ec received from {username='slave'} at 127.0.99.133:36361
I20240503 14:00:03.880546 6680 tablet_copy_service.cc:433] P 2145d15ab2d74c0287839266f1411a43: ending tablet copy session 06ca8dbc004c42b99faa527a4e6f9545-95fb9ad43cf34a78b98965035cd086ec on tablet 95fb9ad43cf34a78b98965035cd086ec with peer 06ca8dbc004c42b99faa527a4e6f9545
I20240503 14:00:03.896734 6526 raft_consensus.cc:1261] T 3d9906a118744dd6a78d50e52f1961e2 P 5f46e19644fd4f4b825c86ba6af768da [term 3 FOLLOWER]: Refusing update from remote peer 8d574fb4a89d4eea947c859470d44dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 14:00:03.897105 16823 consensus_queue.cc:1035] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:03.905951 12748 raft_consensus.cc:482] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:03.906049 12748 raft_consensus.cc:504] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 14:00:03.906386 12748 leader_election.cc:288] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
W20240503 14:00:03.907202 6858 leader_election.cc:334] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:03.907378 6856 leader_election.cc:334] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:03.907455 6856 leader_election.cc:302] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:03.907577 12748 raft_consensus.cc:2726] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:03.920668 6463 consensus_peers.cc:485] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:03.941208 6465 consensus_peers.cc:485] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:03.943524 6465 consensus_peers.cc:485] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:03.944473 6858 consensus_peers.cc:485] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:03.959705 8882 consensus_peers.cc:485] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:03.964044 6463 consensus_peers.cc:485] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:03.965342 9017 consensus_peers.cc:485] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:03.969686 9146 consensus_peers.cc:485] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:03.989733 6372 catalog_manager.cc:4573] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3856ff74b3bf4f608e3c6cdd06ce144e: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:04.007216 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:04.011444 12063 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:04.019099 6594 consensus_peers.cc:485] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.061576 9146 consensus_peers.cc:485] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.061654 9148 consensus_peers.cc:485] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.092097 9146 consensus_peers.cc:485] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.095424 9015 consensus_peers.cc:485] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.095638 9146 consensus_peers.cc:485] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.096278 6594 consensus_peers.cc:485] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.101423 6856 consensus_peers.cc:485] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.108255 6856 consensus_peers.cc:485] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.130055 6856 consensus_peers.cc:485] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.152374 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:04.155380 8882 consensus_peers.cc:485] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.167474 8882 consensus_peers.cc:485] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.174021 9146 consensus_peers.cc:485] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.191545 6594 consensus_peers.cc:485] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.219178 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:04.223052 9146 consensus_peers.cc:485] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:00:04.228773 16828 raft_consensus.cc:482] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:04.228904 16828 raft_consensus.cc:504] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 14:00:04.229758 16828 leader_election.cc:288] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
W20240503 14:00:04.231981 6858 leader_election.cc:334] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:04.232664 6856 leader_election.cc:334] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:04.232744 6856 leader_election.cc:302] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:04.232959 16828 raft_consensus.cc:2726] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:04.241468 6594 consensus_peers.cc:485] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.252831 6856 consensus_peers.cc:485] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.274205 9015 consensus_peers.cc:485] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.293483 9146 consensus_peers.cc:485] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.299510 6856 consensus_peers.cc:485] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.351946 8882 consensus_peers.cc:485] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:00:04.409482 12063 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:04.417203 6594 consensus_peers.cc:485] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:00:04.431324 8946 raft_consensus.cc:1203] T 95fb9ad43cf34a78b98965035cd086ec P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 LEARNER]: Deduplicated request from leader. Original: 1.4->[2.5-2.5] Dedup: 2.5->[]
W20240503 14:00:04.439204 6594 consensus_peers.cc:485] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.505242 8882 consensus_peers.cc:485] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.549970 6594 consensus_peers.cc:485] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.566555 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:04.575886 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:04.581737 16828 raft_consensus.cc:482] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:04.581858 16828 raft_consensus.cc:504] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 14:00:04.582278 16828 leader_election.cc:288] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
W20240503 14:00:04.583058 6856 leader_election.cc:334] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:04.583653 6858 leader_election.cc:334] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:04.583734 6858 leader_election.cc:302] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:04.583863 16828 raft_consensus.cc:2726] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:04.592450 16819 raft_consensus.cc:482] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:04.592554 16819 raft_consensus.cc:504] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 14:00:04.592883 16819 leader_election.cc:288] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
W20240503 14:00:04.593758 6596 leader_election.cc:334] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:04.593848 6594 leader_election.cc:334] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:04.593926 6594 leader_election.cc:302] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:04.594055 16819 raft_consensus.cc:2726] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:04.628108 9146 consensus_peers.cc:485] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.681748 6596 consensus_peers.cc:485] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.692289 9148 consensus_peers.cc:485] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.738020 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:04.772265 9148 consensus_peers.cc:485] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.856611 6858 consensus_peers.cc:485] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:00:04.911870 12063 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:04.919879 9148 consensus_peers.cc:485] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.941274 6858 consensus_peers.cc:485] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.960870 8884 consensus_peers.cc:485] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.973183 8884 consensus_peers.cc:485] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:04.987547 8884 consensus_peers.cc:485] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:05.012434 6858 consensus_peers.cc:485] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:05.026696 6596 consensus_peers.cc:485] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:05.041306 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:05.043917 16819 raft_consensus.cc:1050] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43: attempting to promote NON_VOTER 06ca8dbc004c42b99faa527a4e6f9545 to VOTER
I20240503 14:00:05.044307 16819 consensus_queue.cc:237] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } }
I20240503 14:00:05.045652 8946 raft_consensus.cc:1261] T 95fb9ad43cf34a78b98965035cd086ec P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 LEARNER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:05.045892 6920 raft_consensus.cc:1261] T 95fb9ad43cf34a78b98965035cd086ec P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Refusing update from remote peer 2145d15ab2d74c0287839266f1411a43: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
W20240503 14:00:05.046133 6594 consensus_peers.cc:485] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:05.046239 16855 consensus_queue.cc:1035] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:05.046406 16856 consensus_queue.cc:1035] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:05.048883 16856 raft_consensus.cc:2932] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 14:00:05.049818 8946 raft_consensus.cc:2932] T 95fb9ad43cf34a78b98965035cd086ec P 06ca8dbc004c42b99faa527a4e6f9545 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
I20240503 14:00:05.052760 6387 catalog_manager.cc:5484] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 reported cstate change: config changed from index 4 to 6, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "2145d15ab2d74c0287839266f1411a43" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:05.052794 6920 raft_consensus.cc:2932] T 95fb9ad43cf34a78b98965035cd086ec P f58065db70b64ef8b1dc499aa6ae4460 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, 06ca8dbc004c42b99faa527a4e6f9545 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "06ca8dbc004c42b99faa527a4e6f9545" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34547 } attrs { promote: false } } }
W20240503 14:00:05.057435 9148 consensus_peers.cc:485] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:05.059564 6596 consensus_peers.cc:485] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:05.082401 9148 consensus_peers.cc:485] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:05.101716 9148 consensus_peers.cc:485] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:05.125855 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:05.238921 6858 consensus_peers.cc:485] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:05.242568 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:05.288473 6858 consensus_peers.cc:485] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:05.313527 6465 consensus_peers.cc:485] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:05.329135 6463 consensus_peers.cc:485] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:05.370119 6465 consensus_peers.cc:485] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:00:05.395345 16864 raft_consensus.cc:482] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:05.395467 16864 raft_consensus.cc:504] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 14:00:05.395911 16864 leader_election.cc:288] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
W20240503 14:00:05.397349 6858 leader_election.cc:334] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:05.397724 6856 leader_election.cc:334] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:05.397799 6856 leader_election.cc:302] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:05.398006 16864 raft_consensus.cc:2726] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:05.416417 6594 consensus_peers.cc:485] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:00:05.515117 12063 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:05.537760 6856 consensus_peers.cc:485] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:05.548228 6856 consensus_peers.cc:485] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:05.551116 8884 consensus_peers.cc:485] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:05.596581 9017 consensus_peers.cc:485] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:05.598779 6465 consensus_peers.cc:485] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:05.599725 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:05.628588 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:05.671672 9017 consensus_peers.cc:485] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:05.675660 9017 consensus_peers.cc:485] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:05.676254 9015 consensus_peers.cc:485] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:05.730517 8884 consensus_peers.cc:485] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:00:05.741928 12748 raft_consensus.cc:482] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:05.742060 12748 raft_consensus.cc:504] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 14:00:05.742425 12748 leader_election.cc:288] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
W20240503 14:00:05.743383 6858 leader_election.cc:334] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:05.744700 6856 leader_election.cc:334] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:05.744781 6856 leader_election.cc:302] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:05.745102 12748 raft_consensus.cc:2726] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:05.754088 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:05.764500 9015 consensus_peers.cc:485] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:05.784058 6858 consensus_peers.cc:485] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:05.791970 9148 consensus_peers.cc:485] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:05.793228 9017 consensus_peers.cc:485] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:05.819882 9015 consensus_peers.cc:485] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:05.831272 6596 consensus_peers.cc:485] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:05.858053 6465 consensus_peers.cc:485] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:05.875000 9017 consensus_peers.cc:485] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:05.898253 9017 consensus_peers.cc:485] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:05.923864 6858 consensus_peers.cc:485] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:05.938299 6594 consensus_peers.cc:485] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:05.939852 6463 consensus_peers.cc:485] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:05.945423 6856 consensus_peers.cc:485] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:05.955273 6856 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111) [suppressed 202 similar messages]
W20240503 14:00:05.955902 6856 consensus_peers.cc:485] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:05.958956 9017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111) [suppressed 173 similar messages]
W20240503 14:00:05.960525 8882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 14:00:05.961108 8882 consensus_peers.cc:485] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:05.977773 6594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111) [suppressed 183 similar messages]
W20240503 14:00:05.978955 9148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111) [suppressed 214 similar messages]
W20240503 14:00:06.002236 6465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 14:00:06.010998 9148 consensus_peers.cc:485] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:00:06.015465 16868 raft_consensus.cc:482] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:06.015810 16868 raft_consensus.cc:504] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 14:00:06.016486 16868 leader_election.cc:288] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
W20240503 14:00:06.017385 6465 leader_election.cc:334] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:06.017477 6463 leader_election.cc:334] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:06.017588 6463 leader_election.cc:302] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:06.017740 16868 raft_consensus.cc:2726] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:06.030788 6463 consensus_peers.cc:485] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:06.048600 6856 consensus_peers.cc:485] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:06.060778 6858 consensus_peers.cc:485] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.088749 6372 catalog_manager.cc:4573] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3856ff74b3bf4f608e3c6cdd06ce144e: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:06.102025 9017 consensus_peers.cc:485] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.109503 6596 consensus_peers.cc:485] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:00:06.113649 12063 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:06.126993 6594 consensus_peers.cc:485] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.128410 6858 consensus_peers.cc:485] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.162308 8884 consensus_peers.cc:485] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.174532 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:06.180269 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:06.196825 6463 consensus_peers.cc:485] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.241516 8882 consensus_peers.cc:485] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:06.271194 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:06.281550 6465 consensus_peers.cc:485] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.288134 9148 consensus_peers.cc:485] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.302034 9146 consensus_peers.cc:485] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.308432 6596 consensus_peers.cc:485] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.331588 6463 consensus_peers.cc:485] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.341908 6858 consensus_peers.cc:485] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.348704 9146 consensus_peers.cc:485] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.358119 6465 consensus_peers.cc:485] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.420785 9148 consensus_peers.cc:485] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.426971 9146 consensus_peers.cc:485] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.436810 6596 consensus_peers.cc:485] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.453680 9017 consensus_peers.cc:485] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.475854 9146 consensus_peers.cc:485] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.484269 9017 consensus_peers.cc:485] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.486948 6856 consensus_peers.cc:485] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.496342 9015 consensus_peers.cc:485] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.506217 9148 consensus_peers.cc:485] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.516852 6594 consensus_peers.cc:485] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.519538 6856 consensus_peers.cc:485] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.540097 6594 consensus_peers.cc:485] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.589967 9015 consensus_peers.cc:485] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.592303 9146 consensus_peers.cc:485] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.597520 9017 consensus_peers.cc:485] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:06.599762 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:06.601656 9146 consensus_peers.cc:485] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.614912 8882 consensus_peers.cc:485] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.647979 8882 consensus_peers.cc:485] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:00:06.685088 16096 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:06.710079 6594 consensus_peers.cc:485] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.722913 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:06.723474 6594 consensus_peers.cc:485] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.732015 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:06.735908 6856 consensus_peers.cc:485] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:00:06.743131 16819 raft_consensus.cc:482] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:06.743233 16819 raft_consensus.cc:504] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 14:00:06.743651 16819 leader_election.cc:288] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
W20240503 14:00:06.744549 6594 leader_election.cc:334] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:06.744794 6596 leader_election.cc:334] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:06.744851 6596 leader_election.cc:302] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:06.744985 16819 raft_consensus.cc:2726] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:06.780835 8882 consensus_peers.cc:485] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.784297 8882 consensus_peers.cc:485] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.836071 9015 consensus_peers.cc:485] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.845160 6856 consensus_peers.cc:485] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.863132 6594 consensus_peers.cc:485] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.905519 9146 consensus_peers.cc:485] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.914822 9146 consensus_peers.cc:485] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.931428 6596 consensus_peers.cc:485] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:06.997387 8882 consensus_peers.cc:485] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:07.068399 9146 consensus_peers.cc:485] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:07.069372 6856 consensus_peers.cc:485] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:07.072223 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:07.072676 6594 consensus_peers.cc:485] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:00:07.106580 16864 raft_consensus.cc:482] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:07.106690 16864 raft_consensus.cc:504] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 14:00:07.107069 16864 leader_election.cc:288] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
W20240503 14:00:07.108088 6858 leader_election.cc:334] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:07.108166 6856 leader_election.cc:334] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:07.108249 6856 leader_election.cc:302] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:07.108366 16864 raft_consensus.cc:2726] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:07.126906 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:07.140386 6594 consensus_peers.cc:485] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:00:07.148655 16819 raft_consensus.cc:482] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:07.149253 16819 raft_consensus.cc:504] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
W20240503 14:00:07.150621 6594 leader_election.cc:334] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:07.150697 6596 leader_election.cc:334] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:07.150758 6596 leader_election.cc:302] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:07.150907 16852 raft_consensus.cc:2726] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:00:07.151333 16819 leader_election.cc:288] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
W20240503 14:00:07.199044 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:07.234211 9148 consensus_peers.cc:485] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:07.245986 9146 consensus_peers.cc:485] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:00:07.281831 12063 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:07.332718 6858 consensus_peers.cc:485] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:07.333285 6594 consensus_peers.cc:485] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:07.364704 9148 consensus_peers.cc:485] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:07.373297 8884 consensus_peers.cc:485] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:07.420831 9148 consensus_peers.cc:485] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:07.427620 6858 consensus_peers.cc:485] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:07.495868 8884 consensus_peers.cc:485] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:07.541915 6596 consensus_peers.cc:485] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:07.547326 6858 consensus_peers.cc:485] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:07.558301 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:07.586019 16877 raft_consensus.cc:482] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:07.586146 16877 raft_consensus.cc:504] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 14:00:07.586571 16877 leader_election.cc:288] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
W20240503 14:00:07.587673 6856 leader_election.cc:334] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:07.587814 6858 leader_election.cc:334] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:07.587899 6858 leader_election.cc:302] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:07.588023 16877 raft_consensus.cc:2726] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:07.593149 9148 consensus_peers.cc:485] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:00:07.604584 16877 raft_consensus.cc:482] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:07.604744 16877 raft_consensus.cc:504] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 14:00:07.605118 16877 leader_election.cc:288] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
W20240503 14:00:07.606060 6856 leader_election.cc:334] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:07.606141 6858 leader_election.cc:334] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:07.606202 6858 leader_election.cc:302] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:07.606369 16877 raft_consensus.cc:2726] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:07.643528 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:07.648301 8884 consensus_peers.cc:485] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:07.728375 6465 consensus_peers.cc:485] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:07.730039 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:07.774444 9148 consensus_peers.cc:485] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:07.781630 9148 consensus_peers.cc:485] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:00:07.789219 16881 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:07.793907 6858 consensus_peers.cc:485] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:00:07.805352 16875 raft_consensus.cc:482] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:07.805488 16875 raft_consensus.cc:504] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 14:00:07.805888 16875 leader_election.cc:288] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
W20240503 14:00:07.806972 6858 leader_election.cc:334] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:07.807098 6856 leader_election.cc:334] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:07.807157 6856 leader_election.cc:302] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:07.807317 16875 raft_consensus.cc:2726] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:07.816212 6596 consensus_peers.cc:485] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:07.862257 6856 consensus_peers.cc:485] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:07.932550 6594 consensus_peers.cc:485] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:07.957348 9015 consensus_peers.cc:485] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:00:08.020313 16878 raft_consensus.cc:482] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:08.020462 16878 raft_consensus.cc:504] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 14:00:08.020923 16878 leader_election.cc:288] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
W20240503 14:00:08.021900 6465 leader_election.cc:334] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:08.022127 6858 consensus_peers.cc:485] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:08.022334 6463 leader_election.cc:334] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:08.022408 6463 leader_election.cc:302] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:08.022532 16878 raft_consensus.cc:2726] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:08.040498 6465 consensus_peers.cc:485] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:08.069794 9017 consensus_peers.cc:485] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.078464 8884 consensus_peers.cc:485] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.096139 6856 consensus_peers.cc:485] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:08.100811 8884 consensus_peers.cc:485] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.137544 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:08.159894 6858 consensus_peers.cc:485] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.162338 9017 consensus_peers.cc:485] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.182207 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:08.187995 6596 consensus_peers.cc:485] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:08.193413 6463 consensus_peers.cc:485] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:08.236150 6465 consensus_peers.cc:485] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.236887 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:08.250205 9015 consensus_peers.cc:485] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:00:08.308682 16881 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:08.387465 6858 consensus_peers.cc:485] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:08.398927 6856 consensus_peers.cc:485] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:08.417328 9017 consensus_peers.cc:485] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.431892 9015 consensus_peers.cc:485] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:08.457024 9017 consensus_peers.cc:485] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.501155 8882 consensus_peers.cc:485] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:08.501621 9148 consensus_peers.cc:485] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.515337 9017 consensus_peers.cc:485] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.533074 9148 consensus_peers.cc:485] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.538760 6465 consensus_peers.cc:485] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.545222 6594 consensus_peers.cc:485] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:08.552796 9017 consensus_peers.cc:485] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.575196 6856 consensus_peers.cc:485] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:08.602937 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:08.609896 6856 consensus_peers.cc:485] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:08.615541 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:08.619032 8884 consensus_peers.cc:485] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.635368 6858 consensus_peers.cc:485] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.636869 6596 consensus_peers.cc:485] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.643580 9146 consensus_peers.cc:485] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.680076 9148 consensus_peers.cc:485] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.681123 6463 consensus_peers.cc:485] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:08.683244 6463 consensus_peers.cc:485] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:08.711802 9146 consensus_peers.cc:485] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.739425 6858 consensus_peers.cc:485] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.756711 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:08.760653 9148 consensus_peers.cc:485] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.762836 6465 consensus_peers.cc:485] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.789634 9146 consensus_peers.cc:485] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.817911 8882 consensus_peers.cc:485] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:08.847865 6596 consensus_peers.cc:485] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.858299 6463 consensus_peers.cc:485] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.861464 9017 consensus_peers.cc:485] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:00:08.862550 12063 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:08.872783 6465 consensus_peers.cc:485] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.873414 9146 consensus_peers.cc:485] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.879880 6594 consensus_peers.cc:485] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.902063 9017 consensus_peers.cc:485] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.912842 9015 consensus_peers.cc:485] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.934024 6858 consensus_peers.cc:485] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.955821 9146 consensus_peers.cc:485] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.964874 6463 consensus_peers.cc:485] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.980912 6594 consensus_peers.cc:485] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:08.993712 9015 consensus_peers.cc:485] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.011457 9148 consensus_peers.cc:485] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.020309 6594 consensus_peers.cc:485] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.031781 6856 consensus_peers.cc:485] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.071915 8882 consensus_peers.cc:485] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.105662 6856 consensus_peers.cc:485] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.117018 6594 consensus_peers.cc:485] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.118731 6856 consensus_peers.cc:485] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.130545 6594 consensus_peers.cc:485] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.131700 6856 consensus_peers.cc:485] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.135691 8882 consensus_peers.cc:485] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.138077 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:09.150081 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:09.163805 9017 consensus_peers.cc:485] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:09.185402 9146 consensus_peers.cc:485] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.205401 8882 consensus_peers.cc:485] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.249320 9146 consensus_peers.cc:485] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.254848 9146 consensus_peers.cc:485] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.262789 9017 consensus_peers.cc:485] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.265130 9015 consensus_peers.cc:485] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:00:09.267259 12063 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:09.275664 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:09.358646 6594 consensus_peers.cc:485] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.414211 6596 consensus_peers.cc:485] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.457290 6596 consensus_peers.cc:485] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.480573 6594 consensus_peers.cc:485] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.507382 6856 consensus_peers.cc:485] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.511695 6858 consensus_peers.cc:485] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.581492 8882 consensus_peers.cc:485] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.581585 8882 consensus_peers.cc:485] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.603315 9146 consensus_peers.cc:485] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.669449 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:09.676024 16881 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:09.677301 9146 consensus_peers.cc:485] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.696339 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:09.720459 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:09.792148 9148 consensus_peers.cc:485] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.794411 9148 consensus_peers.cc:485] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.803738 9148 consensus_peers.cc:485] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.808519 6594 consensus_peers.cc:485] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:09.837241 6858 consensus_peers.cc:485] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.896792 6596 consensus_peers.cc:485] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.930907 6594 consensus_peers.cc:485] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:09.962680 8884 consensus_peers.cc:485] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:10.030594 8884 consensus_peers.cc:485] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:10.078907 9148 consensus_peers.cc:485] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:10.114231 9148 consensus_peers.cc:485] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:10.116897 6858 consensus_peers.cc:485] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:10.151893 8884 consensus_peers.cc:485] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:10.177870 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:10.185318 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:10.206923 6372 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240503 14:00:10.207777 6372 catalog_manager.cc:4573] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3856ff74b3bf4f608e3c6cdd06ce144e: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:10.264858 6594 consensus_peers.cc:485] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:10.268994 9015 consensus_peers.cc:485] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:10.269729 6465 consensus_peers.cc:485] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:00:10.278857 16892 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:10.287297 6465 consensus_peers.cc:485] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:10.335268 6856 consensus_peers.cc:485] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:10.340154 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:10.390528 9148 consensus_peers.cc:485] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:10.392697 6858 consensus_peers.cc:485] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:10.429736 6858 consensus_peers.cc:485] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:10.467499 6596 consensus_peers.cc:485] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:10.474938 6463 consensus_peers.cc:485] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:00:10.514765 16852 raft_consensus.cc:482] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:10.514880 16852 raft_consensus.cc:504] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 14:00:10.515301 16852 leader_election.cc:288] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
W20240503 14:00:10.516185 6596 leader_election.cc:334] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:10.516589 6594 leader_election.cc:334] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:10.516650 6594 leader_election.cc:302] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:10.516799 16852 raft_consensus.cc:2726] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:10.521142 6856 consensus_peers.cc:485] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:10.567157 6596 consensus_peers.cc:485] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:10.571094 8884 consensus_peers.cc:485] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:10.595255 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:10.623554 9017 consensus_peers.cc:485] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:10.693703 6465 consensus_peers.cc:485] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:10.724280 8884 consensus_peers.cc:485] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:10.738500 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:10.782052 9015 consensus_peers.cc:485] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:00:10.789147 16892 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:10.797029 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:10.797511 6858 consensus_peers.cc:485] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:10.806766 9017 consensus_peers.cc:485] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:10.820600 8884 consensus_peers.cc:485] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:10.865496 9148 consensus_peers.cc:485] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:10.897089 8882 consensus_peers.cc:485] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:10.901495 6858 consensus_peers.cc:485] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:10.910316 9015 consensus_peers.cc:485] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:10.910913 6858 consensus_peers.cc:485] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:10.946655 9017 consensus_peers.cc:485] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:10.958060 9017 consensus_peers.cc:485] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:10.962641 6858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111) [suppressed 212 similar messages]
W20240503 14:00:10.978364 6594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111) [suppressed 193 similar messages]
W20240503 14:00:10.988461 9017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111) [suppressed 167 similar messages]
W20240503 14:00:10.989053 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:11.003561 6463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 14:00:11.004252 6463 consensus_peers.cc:485] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:11.024955 6858 consensus_peers.cc:485] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.030583 8884 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 14:00:11.058001 9148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111) [suppressed 223 similar messages]
W20240503 14:00:11.094658 9017 consensus_peers.cc:485] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.102981 6856 consensus_peers.cc:485] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:11.103096 6856 consensus_peers.cc:485] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:11.116403 6856 consensus_peers.cc:485] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:11.138757 9148 consensus_peers.cc:485] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.151974 6858 consensus_peers.cc:485] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.152997 9017 consensus_peers.cc:485] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.157341 9148 consensus_peers.cc:485] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:00:11.181164 12063 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:11.195703 9146 consensus_peers.cc:485] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.207758 6594 consensus_peers.cc:485] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:11.225258 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:11.251833 6596 consensus_peers.cc:485] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.264143 6465 consensus_peers.cc:485] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.342262 9146 consensus_peers.cc:485] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.345296 6465 consensus_peers.cc:485] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.364836 9015 consensus_peers.cc:485] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.365690 9146 consensus_peers.cc:485] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.378798 6465 consensus_peers.cc:485] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.388403 9148 consensus_peers.cc:485] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.390064 6463 consensus_peers.cc:485] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.400835 6594 consensus_peers.cc:485] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.403374 6856 consensus_peers.cc:485] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.409547 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:11.428988 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:11.435899 8882 consensus_peers.cc:485] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.446131 8882 consensus_peers.cc:485] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:11.447777 9017 consensus_peers.cc:485] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.472729 9015 consensus_peers.cc:485] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.472318 9148 consensus_peers.cc:485] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.504048 6596 consensus_peers.cc:485] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.504920 6594 consensus_peers.cc:485] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.506042 9017 consensus_peers.cc:485] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:11.544018 9146 consensus_peers.cc:485] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.550972 6463 consensus_peers.cc:485] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.561573 9017 consensus_peers.cc:485] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.578949 9146 consensus_peers.cc:485] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.590762 6594 consensus_peers.cc:485] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.619431 6463 consensus_peers.cc:485] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:11.638698 9146 consensus_peers.cc:485] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.654538 6594 consensus_peers.cc:485] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:00:11.662501 16877 raft_consensus.cc:482] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:11.662628 16877 raft_consensus.cc:504] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 14:00:11.663007 16877 leader_election.cc:288] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
W20240503 14:00:11.664017 6856 leader_election.cc:334] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:11.664806 6858 leader_election.cc:334] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:11.664883 6858 leader_election.cc:302] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:11.665141 16875 raft_consensus.cc:2726] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:11.665215 9146 consensus_peers.cc:485] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.702144 6856 consensus_peers.cc:485] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.705133 9015 consensus_peers.cc:485] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.705590 6594 consensus_peers.cc:485] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.707834 6594 consensus_peers.cc:485] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.718887 9146 consensus_peers.cc:485] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:00:11.724143 16875 raft_consensus.cc:482] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:11.724262 16875 raft_consensus.cc:504] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 14:00:11.724679 16875 leader_election.cc:288] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
W20240503 14:00:11.725693 6858 leader_election.cc:334] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:11.725840 6856 leader_election.cc:334] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:11.725924 6856 leader_election.cc:302] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:11.726038 16875 raft_consensus.cc:2726] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:11.741427 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:11.764304 6858 consensus_peers.cc:485] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.765565 8882 consensus_peers.cc:485] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.772662 6856 consensus_peers.cc:485] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:00:11.778048 12063 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:11.790732 8882 consensus_peers.cc:485] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.810010 6594 consensus_peers.cc:485] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:00:11.826470 16852 raft_consensus.cc:482] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:11.826620 16852 raft_consensus.cc:504] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 14:00:11.827157 16852 leader_election.cc:288] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
W20240503 14:00:11.828579 6596 leader_election.cc:334] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:11.828732 6594 leader_election.cc:334] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:11.828814 6594 leader_election.cc:302] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:11.828951 16852 raft_consensus.cc:2726] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:11.841064 9017 consensus_peers.cc:485] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:11.845933 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:11.982033 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:12.005420 6856 consensus_peers.cc:485] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:12.015656 6856 consensus_peers.cc:485] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:12.034592 9148 consensus_peers.cc:485] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:12.086997 8882 consensus_peers.cc:485] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:12.092662 9148 consensus_peers.cc:485] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:12.097929 6596 consensus_peers.cc:485] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:12.108022 9146 consensus_peers.cc:485] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:12.176419 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:12.232451 16892 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:12.234263 6596 consensus_peers.cc:485] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:12.240909 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:12.246131 6858 consensus_peers.cc:485] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:12.281270 8882 consensus_peers.cc:485] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:12.324579 6594 consensus_peers.cc:485] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:12.347082 9148 consensus_peers.cc:485] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:12.349288 6594 consensus_peers.cc:485] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:12.386473 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:12.447326 8884 consensus_peers.cc:485] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:12.465860 9146 consensus_peers.cc:485] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:00:12.487809 16875 raft_consensus.cc:482] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:12.487916 16875 raft_consensus.cc:504] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 14:00:12.488286 16875 leader_election.cc:288] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
W20240503 14:00:12.489156 6858 leader_election.cc:334] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:12.489252 6856 leader_election.cc:334] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:12.489310 6856 leader_election.cc:302] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:12.489419 16875 raft_consensus.cc:2726] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:12.495200 9148 consensus_peers.cc:485] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:12.529537 6596 consensus_peers.cc:485] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:00:12.536167 16900 raft_consensus.cc:482] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:12.536291 16900 raft_consensus.cc:504] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 14:00:12.536690 16900 leader_election.cc:288] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
W20240503 14:00:12.537549 6465 leader_election.cc:334] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:12.537689 6463 leader_election.cc:334] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:12.537762 6463 leader_election.cc:302] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:12.537909 16900 raft_consensus.cc:2726] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:12.555461 6858 consensus_peers.cc:485] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:12.559882 8884 consensus_peers.cc:485] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:12.584697 6465 consensus_peers.cc:485] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:12.594166 8884 consensus_peers.cc:485] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:12.630239 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:12.655880 16875 raft_consensus.cc:482] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:12.656024 16875 raft_consensus.cc:504] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 14:00:12.656404 16875 leader_election.cc:288] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
W20240503 14:00:12.657200 6856 leader_election.cc:334] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:12.657507 6858 leader_election.cc:334] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:12.657577 6858 leader_election.cc:302] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:12.657697 16875 raft_consensus.cc:2726] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:12.683872 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:12.694831 9148 consensus_peers.cc:485] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:00:12.701191 12063 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:12.796032 9148 consensus_peers.cc:485] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:12.824183 6594 consensus_peers.cc:485] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:12.826320 6856 consensus_peers.cc:485] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:12.855908 6465 consensus_peers.cc:485] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:12.869004 6596 consensus_peers.cc:485] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:12.885007 6856 consensus_peers.cc:485] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:12.929169 9017 consensus_peers.cc:485] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:12.933216 6463 consensus_peers.cc:485] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:12.942471 6596 consensus_peers.cc:485] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:12.961621 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:13.017655 6465 consensus_peers.cc:485] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.057927 8884 consensus_peers.cc:485] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.068264 8884 consensus_peers.cc:485] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.079540 9015 consensus_peers.cc:485] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:13.087122 9148 consensus_peers.cc:485] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.125352 6858 consensus_peers.cc:485] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.136449 6858 consensus_peers.cc:485] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:13.160859 8884 consensus_peers.cc:485] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.204564 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:13.246210 9017 consensus_peers.cc:485] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:00:13.263103 12063 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:13.266619 9015 consensus_peers.cc:485] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:13.283605 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:13.289206 8882 consensus_peers.cc:485] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:13.296013 6858 consensus_peers.cc:485] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.306201 9017 consensus_peers.cc:485] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.336061 6463 consensus_peers.cc:485] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:13.379747 9017 consensus_peers.cc:485] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.431351 6858 consensus_peers.cc:485] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.473067 6858 consensus_peers.cc:485] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.518453 6856 consensus_peers.cc:485] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:13.540158 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:13.559031 8882 consensus_peers.cc:485] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:13.590919 9017 consensus_peers.cc:485] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.605568 9148 consensus_peers.cc:485] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.606446 6856 consensus_peers.cc:485] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:13.632931 9017 consensus_peers.cc:485] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:13.633035 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:13.667493 6465 consensus_peers.cc:485] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.675098 9015 consensus_peers.cc:485] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:13.677718 9017 consensus_peers.cc:485] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.677973 9146 consensus_peers.cc:485] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.679574 6594 consensus_peers.cc:485] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.685137 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:13.693209 9146 consensus_peers.cc:485] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.738193 9148 consensus_peers.cc:485] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:00:13.739148 16914 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:13.757778 6594 consensus_peers.cc:485] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:13.763237 6856 consensus_peers.cc:485] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:13.765542 9148 consensus_peers.cc:485] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.775943 6858 consensus_peers.cc:485] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:13.778863 9146 consensus_peers.cc:485] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.792372 9148 consensus_peers.cc:485] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.798939 6465 consensus_peers.cc:485] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.838414 6594 consensus_peers.cc:485] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.865996 6596 consensus_peers.cc:485] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.888931 6463 consensus_peers.cc:485] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.891806 9146 consensus_peers.cc:485] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.895817 6858 consensus_peers.cc:485] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.905768 9017 consensus_peers.cc:485] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.924628 6463 consensus_peers.cc:485] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.937139 6596 consensus_peers.cc:485] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.961571 6465 consensus_peers.cc:485] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.966208 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:13.966924 9015 consensus_peers.cc:485] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.994688 8882 consensus_peers.cc:485] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:13.997489 6594 consensus_peers.cc:485] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.051301 6856 consensus_peers.cc:485] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.053951 9015 consensus_peers.cc:485] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.083551 9015 consensus_peers.cc:485] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.091572 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:14.134064 6594 consensus_peers.cc:485] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.152963 9146 consensus_peers.cc:485] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.154582 9017 consensus_peers.cc:485] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.173107 6594 consensus_peers.cc:485] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.176749 8882 consensus_peers.cc:485] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.177999 9146 consensus_peers.cc:485] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.194046 6463 consensus_peers.cc:485] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:14.201495 9146 consensus_peers.cc:485] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.226253 6594 consensus_peers.cc:485] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.226805 6594 consensus_peers.cc:485] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:00:14.241309 16910 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:14.266252 9146 consensus_peers.cc:485] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.292563 6856 consensus_peers.cc:485] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.294272 6856 consensus_peers.cc:485] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.299446 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:14.363873 6856 consensus_peers.cc:485] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.372823 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:14.391062 9017 consensus_peers.cc:485] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.453835 6858 consensus_peers.cc:485] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.527137 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:14.535547 6856 consensus_peers.cc:485] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.549916 8882 consensus_peers.cc:485] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:00:14.561765 16917 raft_consensus.cc:482] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:14.561905 16917 raft_consensus.cc:504] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 14:00:14.562301 16917 leader_election.cc:288] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
W20240503 14:00:14.563107 6594 leader_election.cc:334] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:14.563190 6596 leader_election.cc:334] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:14.563263 6596 leader_election.cc:302] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:14.563400 16917 raft_consensus.cc:2726] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:14.692456 9148 consensus_peers.cc:485] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.694167 6858 consensus_peers.cc:485] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.707604 8882 consensus_peers.cc:485] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.709288 9148 consensus_peers.cc:485] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.718979 9148 consensus_peers.cc:485] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.764050 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:14.798087 9146 consensus_peers.cc:485] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:00:14.817936 16914 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:14.842761 6596 consensus_peers.cc:485] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.856607 8882 consensus_peers.cc:485] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.896140 6594 consensus_peers.cc:485] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.913621 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:14.936844 9148 consensus_peers.cc:485] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.940214 6596 consensus_peers.cc:485] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:14.941283 8884 consensus_peers.cc:485] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:15.024132 6594 consensus_peers.cc:485] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:15.030827 8884 consensus_peers.cc:485] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:15.033300 6858 consensus_peers.cc:485] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:15.077575 6465 consensus_peers.cc:485] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:15.092384 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:15.104710 9148 consensus_peers.cc:485] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:15.105625 8884 consensus_peers.cc:485] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:15.190302 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:15.215868 9146 consensus_peers.cc:485] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:15.228245 9148 consensus_peers.cc:485] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:15.237999 9015 consensus_peers.cc:485] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:15.244058 6596 consensus_peers.cc:485] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:15.307941 6596 consensus_peers.cc:485] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:15.313373 8884 consensus_peers.cc:485] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:15.329311 6463 consensus_peers.cc:485] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:15.347704 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:15.354919 6856 consensus_peers.cc:485] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:15.363576 6465 consensus_peers.cc:485] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:00:15.383668 16910 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:15.444628 8884 consensus_peers.cc:485] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:15.481998 6858 consensus_peers.cc:485] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:15.499300 6465 consensus_peers.cc:485] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:15.504532 6596 consensus_peers.cc:485] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:15.506873 6856 consensus_peers.cc:485] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:15.517670 6858 consensus_peers.cc:485] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:15.555177 6594 consensus_peers.cc:485] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:15.613950 9017 consensus_peers.cc:485] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:15.643882 8884 consensus_peers.cc:485] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:00:15.653059 16906 raft_consensus.cc:482] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:15.653206 16906 raft_consensus.cc:504] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 14:00:15.653585 16906 leader_election.cc:288] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
W20240503 14:00:15.655335 6594 leader_election.cc:334] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:15.656242 6596 leader_election.cc:334] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:15.656325 6596 leader_election.cc:302] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:15.656509 16906 raft_consensus.cc:2726] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:15.667371 8882 consensus_peers.cc:485] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:15.697739 9148 consensus_peers.cc:485] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:15.704002 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:15.716070 9017 consensus_peers.cc:485] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:15.723497 9015 consensus_peers.cc:485] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:15.751245 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:15.780345 16877 raft_consensus.cc:482] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:15.780491 16877 raft_consensus.cc:504] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 14:00:15.780989 16877 leader_election.cc:288] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
W20240503 14:00:15.781944 6858 leader_election.cc:334] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:15.782119 6856 leader_election.cc:334] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:15.782191 6856 leader_election.cc:302] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:15.782325 16877 raft_consensus.cc:2726] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:15.865341 6858 consensus_peers.cc:485] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:15.872486 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:15.906564 16914 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:15.933733 6858 consensus_peers.cc:485] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:15.969482 6463 consensus_peers.cc:485] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:16.003446 6594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111) [suppressed 191 similar messages]
W20240503 14:00:16.009654 6856 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111) [suppressed 206 similar messages]
W20240503 14:00:16.011791 8882 consensus_peers.cc:485] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:16.014003 9015 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111) [suppressed 167 similar messages]
W20240503 14:00:16.027207 9146 consensus_peers.cc:485] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.047602 9146 consensus_peers.cc:485] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.059140 6856 consensus_peers.cc:485] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:16.060067 9017 consensus_peers.cc:485] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.060528 9015 consensus_peers.cc:485] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:16.069234 9146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111) [suppressed 222 similar messages]
W20240503 14:00:16.075153 9017 consensus_peers.cc:485] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:16.084379 8882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240503 14:00:16.104806 6465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111) [suppressed 113 similar messages]
W20240503 14:00:16.112991 9017 consensus_peers.cc:485] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.118702 9148 consensus_peers.cc:485] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.131388 6856 consensus_peers.cc:485] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:16.135250 9017 consensus_peers.cc:485] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.144569 6858 consensus_peers.cc:485] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.157882 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:16.170789 6594 consensus_peers.cc:485] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.171207 6858 consensus_peers.cc:485] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:16.172490 6594 consensus_peers.cc:485] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.176506 6596 consensus_peers.cc:485] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.185564 6856 consensus_peers.cc:485] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:16.190490 6465 consensus_peers.cc:485] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.217646 9148 consensus_peers.cc:485] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.245968 6463 consensus_peers.cc:485] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.251662 9148 consensus_peers.cc:485] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.252172 9017 consensus_peers.cc:485] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.252375 6465 consensus_peers.cc:485] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.255597 6463 consensus_peers.cc:485] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.268771 6594 consensus_peers.cc:485] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:16.280766 9148 consensus_peers.cc:485] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.340941 9146 consensus_peers.cc:485] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:00:16.355247 16910 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:16.364758 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:16.383478 9015 consensus_peers.cc:485] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.414342 9146 consensus_peers.cc:485] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.424175 6858 consensus_peers.cc:485] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.427397 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:16.492779 9015 consensus_peers.cc:485] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.519457 6594 consensus_peers.cc:485] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.523891 9146 consensus_peers.cc:485] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.539546 9146 consensus_peers.cc:485] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.544632 6596 consensus_peers.cc:485] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.544907 8882 consensus_peers.cc:485] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.550365 6463 consensus_peers.cc:485] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:16.559597 6594 consensus_peers.cc:485] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.565418 6858 consensus_peers.cc:485] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.575253 6594 consensus_peers.cc:485] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.619853 6856 consensus_peers.cc:485] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.630548 6856 consensus_peers.cc:485] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.651001 9015 consensus_peers.cc:485] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.656739 8882 consensus_peers.cc:485] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.656926 9146 consensus_peers.cc:485] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.658463 6594 consensus_peers.cc:485] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.661929 9017 consensus_peers.cc:485] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.690157 6594 consensus_peers.cc:485] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.733582 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:16.751751 9017 consensus_peers.cc:485] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.755781 9017 consensus_peers.cc:485] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.768338 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:16.777328 6856 consensus_peers.cc:485] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.814929 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:16.817296 6465 consensus_peers.cc:485] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.842365 9146 consensus_peers.cc:485] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:00:16.889866 16910 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:16.922200 9148 consensus_peers.cc:485] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:16.938696 6856 consensus_peers.cc:485] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:17.036162 6856 consensus_peers.cc:485] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:17.068996 9146 consensus_peers.cc:485] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:17.083813 6858 consensus_peers.cc:485] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:00:17.129099 16877 raft_consensus.cc:482] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:17.129236 16877 raft_consensus.cc:504] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 14:00:17.129608 16877 leader_election.cc:288] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
W20240503 14:00:17.130518 6858 leader_election.cc:334] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:17.130609 6856 leader_election.cc:334] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:17.130678 6856 leader_election.cc:302] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:17.130808 16877 raft_consensus.cc:2726] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:17.137248 6596 consensus_peers.cc:485] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:17.139585 8882 consensus_peers.cc:485] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:17.142385 9148 consensus_peers.cc:485] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:17.188756 9148 consensus_peers.cc:485] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:17.242825 9148 consensus_peers.cc:485] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:17.253682 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:17.284605 8882 consensus_peers.cc:485] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:00:17.304706 16914 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:17.314764 8884 consensus_peers.cc:485] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:17.330854 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:17.333220 9148 consensus_peers.cc:485] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:17.383605 8884 consensus_peers.cc:485] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:17.388935 6858 consensus_peers.cc:485] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:17.396577 9148 consensus_peers.cc:485] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:17.420944 6465 consensus_peers.cc:485] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:17.439163 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:17.483043 8882 consensus_peers.cc:485] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:17.553807 6596 consensus_peers.cc:485] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:17.634095 8884 consensus_peers.cc:485] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:17.659842 9146 consensus_peers.cc:485] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:17.691963 8884 consensus_peers.cc:485] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:17.698278 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:17.711356 6594 consensus_peers.cc:485] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:17.766778 6596 consensus_peers.cc:485] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:17.785002 6596 consensus_peers.cc:485] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:17.809140 8884 consensus_peers.cc:485] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:17.814668 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:17.837491 6465 consensus_peers.cc:485] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:00:17.870594 16910 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:17.882097 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:17.890148 6465 consensus_peers.cc:485] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:17.901350 6463 consensus_peers.cc:485] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:17.906145 6596 consensus_peers.cc:485] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:17.939262 6594 consensus_peers.cc:485] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:17.942400 9015 consensus_peers.cc:485] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:17.975788 6858 consensus_peers.cc:485] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:17.978163 9148 consensus_peers.cc:485] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:17.996754 6856 consensus_peers.cc:485] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:18.061756 6594 consensus_peers.cc:485] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:18.074492 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:18.097071 6856 consensus_peers.cc:485] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:18.125248 9017 consensus_peers.cc:485] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.126473 9015 consensus_peers.cc:485] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:18.228480 6858 consensus_peers.cc:485] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.250308 6858 consensus_peers.cc:485] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.262588 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:18.287256 6858 consensus_peers.cc:485] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:18.289921 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:18.305173 8882 consensus_peers.cc:485] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:18.413790 6372 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:18.414533 6372 catalog_manager.cc:4573] TS 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3856ff74b3bf4f608e3c6cdd06ce144e: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:18.417615 9017 consensus_peers.cc:485] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:00:18.426695 16910 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:18.439240 9017 consensus_peers.cc:485] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:00:18.439529 16877 raft_consensus.cc:482] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:18.439640 16877 raft_consensus.cc:504] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 14:00:18.440096 16877 leader_election.cc:288] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
W20240503 14:00:18.440891 6858 leader_election.cc:334] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:18.441154 6856 leader_election.cc:334] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:18.441226 6856 leader_election.cc:302] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:18.441370 16877 raft_consensus.cc:2726] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:18.461985 9146 consensus_peers.cc:485] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.475356 6858 consensus_peers.cc:485] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.503523 8882 consensus_peers.cc:485] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:18.515631 9015 consensus_peers.cc:485] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:18.526876 6465 consensus_peers.cc:485] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.561973 9017 consensus_peers.cc:485] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.566241 6596 consensus_peers.cc:485] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.567613 6463 consensus_peers.cc:485] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:18.594121 8884 consensus_peers.cc:485] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.595005 9146 consensus_peers.cc:485] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.623185 9017 consensus_peers.cc:485] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.632371 6856 consensus_peers.cc:485] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:18.639312 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:18.642968 6856 consensus_peers.cc:485] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:18.652310 9015 consensus_peers.cc:485] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.662407 6594 consensus_peers.cc:485] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.688366 6858 consensus_peers.cc:485] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:18.708451 6465 consensus_peers.cc:485] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.716024 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:18.769098 9146 consensus_peers.cc:485] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.778410 9148 consensus_peers.cc:485] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.793871 6463 consensus_peers.cc:485] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.803131 6856 consensus_peers.cc:485] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:18.824609 6858 consensus_peers.cc:485] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.830161 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:18.840684 9148 consensus_peers.cc:485] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.852614 6856 consensus_peers.cc:485] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.856930 9148 consensus_peers.cc:485] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:00:18.868634 16910 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:18.872097 6463 consensus_peers.cc:485] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:18.892987 9017 consensus_peers.cc:485] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.893093 6463 consensus_peers.cc:485] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.896800 9015 consensus_peers.cc:485] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.913739 6594 consensus_peers.cc:485] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.919740 6594 consensus_peers.cc:485] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:18.920897 6594 consensus_peers.cc:485] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.923115 8882 consensus_peers.cc:485] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.932545 9148 consensus_peers.cc:485] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.937215 9146 consensus_peers.cc:485] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.940315 9017 consensus_peers.cc:485] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:18.950608 9146 consensus_peers.cc:485] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:18.955965 9017 consensus_peers.cc:485] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:00:19.046420 16875 raft_consensus.cc:482] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:19.046563 16875 raft_consensus.cc:504] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 14:00:19.046975 16875 leader_election.cc:288] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
W20240503 14:00:19.048058 6858 leader_election.cc:334] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:19.048916 6856 leader_election.cc:334] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:19.049021 6856 leader_election.cc:302] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:19.049158 16875 raft_consensus.cc:2726] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:19.059979 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:19.067909 6856 consensus_peers.cc:485] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.093142 9017 consensus_peers.cc:485] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.115370 6594 consensus_peers.cc:485] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.119256 6594 consensus_peers.cc:485] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.152138 9146 consensus_peers.cc:485] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.165782 9146 consensus_peers.cc:485] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.168073 9146 consensus_peers.cc:485] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.172989 6858 consensus_peers.cc:485] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.190338 9017 consensus_peers.cc:485] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.232997 8882 consensus_peers.cc:485] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.253808 9148 consensus_peers.cc:485] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.275812 6596 consensus_peers.cc:485] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.294780 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:19.312880 6594 consensus_peers.cc:485] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.335340 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:19.351127 6594 consensus_peers.cc:485] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.353492 6856 consensus_peers.cc:485] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.403376 9015 consensus_peers.cc:485] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.411023 6856 consensus_peers.cc:485] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.415036 6465 consensus_peers.cc:485] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:00:19.419348 16914 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:19.553130 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:19.561945 6465 consensus_peers.cc:485] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:19.605356 8882 consensus_peers.cc:485] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.625332 9148 consensus_peers.cc:485] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:00:19.651654 16906 raft_consensus.cc:482] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:19.651794 16906 raft_consensus.cc:504] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 14:00:19.652143 16906 leader_election.cc:288] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
W20240503 14:00:19.653191 6594 leader_election.cc:334] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:19.653270 6596 leader_election.cc:334] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:19.653321 6596 leader_election.cc:302] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:19.653510 16906 raft_consensus.cc:2726] T 4a1398d983c54a7498257f766052e8e9 P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:19.662678 8884 consensus_peers.cc:485] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.695637 6596 consensus_peers.cc:485] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.713508 9148 consensus_peers.cc:485] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.721566 8884 consensus_peers.cc:485] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.724702 9148 consensus_peers.cc:485] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.731812 6858 consensus_peers.cc:485] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.765697 9146 consensus_peers.cc:485] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.810545 6858 consensus_peers.cc:485] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.843078 6856 consensus_peers.cc:485] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.884665 6596 consensus_peers.cc:485] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.897073 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:19.900398 9146 consensus_peers.cc:485] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:00:19.907819 16947 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:19.908036 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:19.941231 6596 consensus_peers.cc:485] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:19.979954 9148 consensus_peers.cc:485] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:20.021725 8882 consensus_peers.cc:485] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:20.058084 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:20.106828 8882 consensus_peers.cc:485] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:20.115026 9148 consensus_peers.cc:485] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:20.164413 6594 consensus_peers.cc:485] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:20.194161 8884 consensus_peers.cc:485] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:20.282613 8884 consensus_peers.cc:485] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:20.330338 6465 consensus_peers.cc:485] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:20.332967 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:20.343878 8884 consensus_peers.cc:485] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:20.382036 6596 consensus_peers.cc:485] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:00:20.401706 16875 raft_consensus.cc:482] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:20.401818 16875 raft_consensus.cc:504] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } attrs { promote: false } }
I20240503 14:00:20.402217 16875 leader_election.cc:288] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
W20240503 14:00:20.403075 6856 leader_election.cc:334] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:20.403309 6858 leader_election.cc:334] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:20.403378 6858 leader_election.cc:302] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:20.403525 16875 raft_consensus.cc:2726] T 1be8960eed8148f5958f6bb412c30a7e P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:20.447726 6463 consensus_peers.cc:485] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:20.454994 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:20.468432 6858 consensus_peers.cc:485] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:00:20.504850 16947 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:20.508884 6465 consensus_peers.cc:485] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:20.510191 6596 consensus_peers.cc:485] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:20.519107 9015 consensus_peers.cc:485] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:20.532074 9148 consensus_peers.cc:485] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:20.536407 6856 consensus_peers.cc:485] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:20.540381 6856 consensus_peers.cc:485] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:20.572804 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:20.589309 6594 consensus_peers.cc:485] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:20.650756 6858 consensus_peers.cc:485] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:20.666972 6858 consensus_peers.cc:485] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:20.763219 6594 consensus_peers.cc:485] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:20.765959 9015 consensus_peers.cc:485] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:20.810217 8882 consensus_peers.cc:485] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:20.832412 9148 consensus_peers.cc:485] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:20.834977 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:20.841327 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:20.886544 9017 consensus_peers.cc:485] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:20.895450 6858 consensus_peers.cc:485] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:00:20.907272 16914 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:20.920130 6596 consensus_peers.cc:485] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:20.936699 9146 consensus_peers.cc:485] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:20.947444 9017 consensus_peers.cc:485] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:20.961120 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:20.973227 8884 consensus_peers.cc:485] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:20.999630 9015 consensus_peers.cc:485] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.000581 9015 consensus_peers.cc:485] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:21.001973 8882 consensus_peers.cc:485] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:21.012490 6856 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111) [suppressed 207 similar messages]
W20240503 14:00:21.017621 9015 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111) [suppressed 171 similar messages]
W20240503 14:00:21.024832 6594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111) [suppressed 193 similar messages]
W20240503 14:00:21.035027 9146 consensus_peers.cc:485] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.086673 9017 consensus_peers.cc:485] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.088851 9148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111) [suppressed 227 similar messages]
W20240503 14:00:21.105291 8882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240503 14:00:21.138120 9146 consensus_peers.cc:485] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.158001 9017 consensus_peers.cc:485] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.163561 6856 consensus_peers.cc:485] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:21.167204 6465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111) [suppressed 108 similar messages]
I20240503 14:00:21.176273 16906 raft_consensus.cc:482] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:21.176391 16906 raft_consensus.cc:504] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 14:00:21.176776 16906 leader_election.cc:288] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
W20240503 14:00:21.177842 6594 leader_election.cc:334] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:21.178021 6596 leader_election.cc:334] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:21.178095 6596 leader_election.cc:302] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:21.178225 16906 raft_consensus.cc:2726] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:21.179203 6463 consensus_peers.cc:485] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:21.181756 6594 consensus_peers.cc:485] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.190613 6465 consensus_peers.cc:485] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.196484 9017 consensus_peers.cc:485] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.201437 6858 consensus_peers.cc:485] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:21.209127 9017 consensus_peers.cc:485] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.220106 6594 consensus_peers.cc:485] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:21.226764 6463 consensus_peers.cc:485] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.239517 6858 consensus_peers.cc:485] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.250238 6856 consensus_peers.cc:485] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:21.255141 6463 consensus_peers.cc:485] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:21.263495 9017 consensus_peers.cc:485] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:00:21.294692 16947 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:21.307391 6858 consensus_peers.cc:485] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.313460 6465 consensus_peers.cc:485] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.327909 6594 consensus_peers.cc:485] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.332659 9146 consensus_peers.cc:485] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.362432 9146 consensus_peers.cc:485] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.367136 6594 consensus_peers.cc:485] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.380414 9148 consensus_peers.cc:485] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.396697 9148 consensus_peers.cc:485] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.403465 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:21.408560 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:21.410125 9015 consensus_peers.cc:485] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.420307 9146 consensus_peers.cc:485] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.449409 9148 consensus_peers.cc:485] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.449525 9146 consensus_peers.cc:485] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.449617 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:21.455094 6856 consensus_peers.cc:485] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.455813 6856 consensus_peers.cc:485] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:21.482756 9017 consensus_peers.cc:485] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.527266 9017 consensus_peers.cc:485] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:21.535341 6594 consensus_peers.cc:485] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.549659 6594 consensus_peers.cc:485] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.555472 8882 consensus_peers.cc:485] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.615974 6858 consensus_peers.cc:485] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.668625 6463 consensus_peers.cc:485] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.674245 9148 consensus_peers.cc:485] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:00:21.725049 16914 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:21.782471 9146 consensus_peers.cc:485] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.812660 6856 consensus_peers.cc:485] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.824682 6856 consensus_peers.cc:485] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.838299 6856 consensus_peers.cc:485] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.842110 9148 consensus_peers.cc:485] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.884999 9015 consensus_peers.cc:485] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.889086 6596 consensus_peers.cc:485] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.892483 8882 consensus_peers.cc:485] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.894435 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:21.896890 9017 consensus_peers.cc:485] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.927464 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:21.985231 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:21.986971 6594 consensus_peers.cc:485] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:21.991487 8884 consensus_peers.cc:485] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:22.003782 6594 consensus_peers.cc:485] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:22.043206 6465 consensus_peers.cc:485] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:22.070967 6596 consensus_peers.cc:485] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:22.098464 9146 consensus_peers.cc:485] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:22.111945 9148 consensus_peers.cc:485] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:22.116925 6856 consensus_peers.cc:485] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:22.152463 6858 consensus_peers.cc:485] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:22.195571 9148 consensus_peers.cc:485] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:22.236864 8882 consensus_peers.cc:485] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:22.255671 6465 consensus_peers.cc:485] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:22.264914 6858 consensus_peers.cc:485] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:00:22.265803 16962 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:22.303577 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:22.325400 9146 consensus_peers.cc:485] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:00:22.349640 16933 raft_consensus.cc:482] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:22.349761 16933 raft_consensus.cc:504] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "5f46e19644fd4f4b825c86ba6af768da" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38021 } }
I20240503 14:00:22.350096 16933 leader_election.cc:288] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
W20240503 14:00:22.351009 6463 leader_election.cc:334] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:22.351156 6465 leader_election.cc:334] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:22.351217 6465 leader_election.cc:302] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f46e19644fd4f4b825c86ba6af768da; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:22.351466 16933 raft_consensus.cc:2726] T 9ac9791abf2b4a60ab7a6e4e2d4b0de6 P 5f46e19644fd4f4b825c86ba6af768da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:22.439222 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:22.470849 6594 consensus_peers.cc:485] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:22.478382 8884 consensus_peers.cc:485] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:22.517120 6596 consensus_peers.cc:485] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:22.543324 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:22.547164 6596 consensus_peers.cc:485] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:22.595671 9148 consensus_peers.cc:485] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:22.657245 8884 consensus_peers.cc:485] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:22.684485 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:22.733736 9148 consensus_peers.cc:485] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:00:22.776139 16962 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:22.781751 6465 consensus_peers.cc:485] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:22.800087 8882 consensus_peers.cc:485] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:22.837090 6596 consensus_peers.cc:485] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:00:22.839390 16906 raft_consensus.cc:482] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:22.839502 16906 raft_consensus.cc:504] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 14:00:22.840130 16906 leader_election.cc:288] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
W20240503 14:00:22.840790 6596 leader_election.cc:334] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:22.840924 6594 leader_election.cc:334] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:22.841012 6594 leader_election.cc:302] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:22.841140 16906 raft_consensus.cc:2726] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:22.844512 8884 consensus_peers.cc:485] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:22.875761 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:22.876286 8882 consensus_peers.cc:485] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:22.903304 6858 consensus_peers.cc:485] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:22.951507 6856 consensus_peers.cc:485] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:22.963745 8884 consensus_peers.cc:485] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.007863 6596 consensus_peers.cc:485] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:23.009533 9148 consensus_peers.cc:485] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.019655 6463 consensus_peers.cc:485] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:23.026760 9015 consensus_peers.cc:485] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:23.096982 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:23.135624 6856 consensus_peers.cc:485] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:00:23.154902 16962 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:23.159111 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:23.178822 6858 consensus_peers.cc:485] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.181481 6594 consensus_peers.cc:485] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:23.196106 6465 consensus_peers.cc:485] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:23.280604 6465 consensus_peers.cc:485] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.298957 9015 consensus_peers.cc:485] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:23.342519 8882 consensus_peers.cc:485] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:23.348749 6858 consensus_peers.cc:485] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.354301 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:23.362689 9017 consensus_peers.cc:485] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.366197 9017 consensus_peers.cc:485] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.367720 9146 consensus_peers.cc:485] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.388825 9148 consensus_peers.cc:485] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.402770 6596 consensus_peers.cc:485] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.425012 6465 consensus_peers.cc:485] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.428023 9017 consensus_peers.cc:485] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.481093 9015 consensus_peers.cc:485] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.486280 6594 consensus_peers.cc:485] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:23.524554 9017 consensus_peers.cc:485] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.535930 6594 consensus_peers.cc:485] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:23.544137 9015 consensus_peers.cc:485] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:23.546886 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:23.562402 6858 consensus_peers.cc:485] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:23.585043 6856 consensus_peers.cc:485] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:23.586236 9146 consensus_peers.cc:485] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.639111 9146 consensus_peers.cc:485] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.696413 9148 consensus_peers.cc:485] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.701301 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:23.707082 9146 consensus_peers.cc:485] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.707500 6463 consensus_peers.cc:485] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:23.739493 6858 consensus_peers.cc:485] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.753288 6858 consensus_peers.cc:485] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:23.770947 6858 consensus_peers.cc:485] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:00:23.771852 16962 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:23.774091 8884 consensus_peers.cc:485] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.778223 9017 consensus_peers.cc:485] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.780144 6856 consensus_peers.cc:485] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:23.785192 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:23.812834 9148 consensus_peers.cc:485] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.816408 6594 consensus_peers.cc:485] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.826961 9148 consensus_peers.cc:485] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.829428 6463 consensus_peers.cc:485] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:23.830061 9017 consensus_peers.cc:485] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.836205 8882 consensus_peers.cc:485] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.842116 8882 consensus_peers.cc:485] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:23.849344 9146 consensus_peers.cc:485] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.869025 9017 consensus_peers.cc:485] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.872095 6463 consensus_peers.cc:485] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.881533 9146 consensus_peers.cc:485] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.892001 6594 consensus_peers.cc:485] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.904865 6858 consensus_peers.cc:485] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.926651 6856 consensus_peers.cc:485] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.930459 9015 consensus_peers.cc:485] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:23.972725 6856 consensus_peers.cc:485] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:23.975147 9017 consensus_peers.cc:485] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:24.031319 9017 consensus_peers.cc:485] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:24.035728 6594 consensus_peers.cc:485] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:24.059734 9015 consensus_peers.cc:485] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:24.085975 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:24.123452 6594 consensus_peers.cc:485] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:00:24.164330 16974 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:24.222119 6856 consensus_peers.cc:485] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:24.245882 9148 consensus_peers.cc:485] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:24.248713 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:24.269882 6596 consensus_peers.cc:485] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:24.275367 6594 consensus_peers.cc:485] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:24.283429 9146 consensus_peers.cc:485] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:24.293067 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:24.296497 9146 consensus_peers.cc:485] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:24.308863 8884 consensus_peers.cc:485] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:24.328568 8882 consensus_peers.cc:485] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:24.330338 6856 consensus_peers.cc:485] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:24.383055 9148 consensus_peers.cc:485] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:24.420635 6856 consensus_peers.cc:485] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:24.441931 9146 consensus_peers.cc:485] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:24.455427 6463 consensus_peers.cc:485] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:24.471067 6465 consensus_peers.cc:485] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:24.494199 9017 consensus_peers.cc:485] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:24.520588 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:24.583128 6858 consensus_peers.cc:485] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:24.594024 6856 consensus_peers.cc:485] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:24.619657 6858 consensus_peers.cc:485] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:24.620508 8882 consensus_peers.cc:485] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:24.623065 6594 consensus_peers.cc:485] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:24.632715 9148 consensus_peers.cc:485] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:24.647059 6594 consensus_peers.cc:485] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:24.709185 6596 consensus_peers.cc:485] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:24.750363 6465 consensus_peers.cc:485] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:00:24.759845 16974 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:24.814160 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:24.839037 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:24.848680 9146 consensus_peers.cc:485] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:00:24.849287 16875 raft_consensus.cc:482] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:24.849382 16875 raft_consensus.cc:504] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 14:00:24.849784 16875 leader_election.cc:288] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
W20240503 14:00:24.850993 6856 leader_election.cc:334] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:24.851109 6858 leader_election.cc:334] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:24.851186 6858 leader_election.cc:302] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:24.851344 16875 raft_consensus.cc:2726] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:24.939806 6594 consensus_peers.cc:485] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:24.951607 6596 consensus_peers.cc:485] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:24.974067 9148 consensus_peers.cc:485] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:25.004676 8884 consensus_peers.cc:485] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:25.014230 6465 consensus_peers.cc:485] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:25.113943 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:25.125269 8884 consensus_peers.cc:485] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:25.127897 6596 consensus_peers.cc:485] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:00:25.206308 16974 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:25.216912 6858 consensus_peers.cc:485] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:25.236775 9015 consensus_peers.cc:485] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:25.238041 8884 consensus_peers.cc:485] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:00:25.307611 16875 raft_consensus.cc:482] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:25.307790 16875 raft_consensus.cc:504] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 14:00:25.308218 16875 leader_election.cc:288] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
W20240503 14:00:25.309108 6856 leader_election.cc:334] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:25.309232 6858 leader_election.cc:334] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:25.309280 6858 leader_election.cc:302] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:25.309429 16875 raft_consensus.cc:2726] T 7a25af1e7ae64f4fad8ad82faa1c6eb6 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:25.310839 9148 consensus_peers.cc:485] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:25.313970 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:25.344746 9148 consensus_peers.cc:485] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:25.364715 8884 consensus_peers.cc:485] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:25.372339 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:25.487905 8882 consensus_peers.cc:485] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:25.537214 9148 consensus_peers.cc:485] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:25.584553 6465 consensus_peers.cc:485] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:25.589010 6596 consensus_peers.cc:485] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:25.591986 6594 consensus_peers.cc:485] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:25.609376 6465 consensus_peers.cc:485] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:25.615352 6465 consensus_peers.cc:485] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:00:25.640415 16990 raft_consensus.cc:482] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d77752181e944698e809e325e5937e6)
I20240503 14:00:25.640529 16990 raft_consensus.cc:504] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } attrs { promote: false } }
I20240503 14:00:25.640982 16990 leader_election.cc:288] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931), 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365)
W20240503 14:00:25.642192 6856 leader_election.cc:334] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:25.642444 6858 leader_election.cc:334] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:25.642515 6858 leader_election.cc:302] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:25.642638 16990 raft_consensus.cc:2726] T 1350225b1a2848debca68e1bad5528c5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:25.653949 8882 consensus_peers.cc:485] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:25.658553 6463 consensus_peers.cc:485] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:25.674726 6856 consensus_peers.cc:485] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:25.675195 6858 consensus_peers.cc:485] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:25.680433 6596 consensus_peers.cc:485] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:25.730412 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:25.739112 9017 consensus_peers.cc:485] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:00:25.757964 16974 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:25.768051 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:25.833163 6856 consensus_peers.cc:485] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:25.835093 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:25.861660 6594 consensus_peers.cc:485] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:25.923624 8882 consensus_peers.cc:485] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:25.925599 9015 consensus_peers.cc:485] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:25.926460 9017 consensus_peers.cc:485] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:25.940006 9017 consensus_peers.cc:485] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:25.946691 9148 consensus_peers.cc:485] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.018572 9146 consensus_peers.cc:485] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.025305 6858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111) [suppressed 201 similar messages]
W20240503 14:00:26.030990 6858 consensus_peers.cc:485] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:26.031178 9017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111) [suppressed 169 similar messages]
W20240503 14:00:26.042691 9146 consensus_peers.cc:485] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.042979 9015 consensus_peers.cc:485] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.047495 6596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111) [suppressed 191 similar messages]
W20240503 14:00:26.048277 6596 consensus_peers.cc:485] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.048740 6463 consensus_peers.cc:485] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:26.049882 6594 consensus_peers.cc:485] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:26.090531 6856 consensus_peers.cc:485] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:26.110190 6858 consensus_peers.cc:485] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.110394 8882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240503 14:00:26.119747 9146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111) [suppressed 216 similar messages]
W20240503 14:00:26.120374 9146 consensus_peers.cc:485] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.153568 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:26.170619 9015 consensus_peers.cc:485] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:26.175209 6858 consensus_peers.cc:485] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.177423 9017 consensus_peers.cc:485] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.179595 9148 consensus_peers.cc:485] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.200649 6465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111) [suppressed 116 similar messages]
W20240503 14:00:26.202872 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:26.209726 6463 consensus_peers.cc:485] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.243225 6594 consensus_peers.cc:485] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.262773 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:26.266469 6856 consensus_peers.cc:485] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:26.277467 6856 consensus_peers.cc:485] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.281837 6858 consensus_peers.cc:485] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.294399 6463 consensus_peers.cc:485] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:26.304484 8882 consensus_peers.cc:485] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.304692 9148 consensus_peers.cc:485] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.322330 6858 consensus_peers.cc:485] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:26.323618 8882 consensus_peers.cc:485] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:26.325162 6594 consensus_peers.cc:485] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.341887 9017 consensus_peers.cc:485] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:00:26.350023 16993 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:26.361948 9146 consensus_peers.cc:485] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.381548 9017 consensus_peers.cc:485] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.381907 9148 consensus_peers.cc:485] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.403256 9146 consensus_peers.cc:485] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.407658 9146 consensus_peers.cc:485] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.413645 9015 consensus_peers.cc:485] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.413837 9017 consensus_peers.cc:485] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.513473 8884 consensus_peers.cc:485] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.550482 6858 consensus_peers.cc:485] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.550994 9017 consensus_peers.cc:485] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.616180 6594 consensus_peers.cc:485] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.647841 9017 consensus_peers.cc:485] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:26.678103 9015 consensus_peers.cc:485] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.723199 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:26.730837 16993 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:26.738408 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:26.776321 6594 consensus_peers.cc:485] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.776810 6594 consensus_peers.cc:485] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.780431 6856 consensus_peers.cc:485] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:26.832769 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:26.855988 6856 consensus_peers.cc:485] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.865468 6594 consensus_peers.cc:485] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.866704 9148 consensus_peers.cc:485] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.867450 9146 consensus_peers.cc:485] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.890239 9017 consensus_peers.cc:485] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.910601 6465 consensus_peers.cc:485] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.931093 6856 consensus_peers.cc:485] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.957715 9148 consensus_peers.cc:485] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.959944 6463 consensus_peers.cc:485] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.983565 6596 consensus_peers.cc:485] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:26.987669 9146 consensus_peers.cc:485] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:27.002010 8884 consensus_peers.cc:485] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:27.020875 6858 consensus_peers.cc:485] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:27.022902 9146 consensus_peers.cc:485] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:27.025307 6858 consensus_peers.cc:485] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:27.078497 6856 consensus_peers.cc:485] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:27.095356 6465 consensus_peers.cc:485] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:27.114238 8882 consensus_peers.cc:485] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:27.139492 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:27.152865 6596 consensus_peers.cc:485] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:27.177743 6594 consensus_peers.cc:485] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:27.186645 8882 consensus_peers.cc:485] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:00:27.200891 16993 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:27.226338 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:27.244580 9148 consensus_peers.cc:485] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:27.259347 6856 consensus_peers.cc:485] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:27.296545 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:27.315505 6594 consensus_peers.cc:485] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:27.393947 8884 consensus_peers.cc:485] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:27.394801 8884 consensus_peers.cc:485] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:27.432077 6596 consensus_peers.cc:485] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:27.441829 9146 consensus_peers.cc:485] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:27.451496 6596 consensus_peers.cc:485] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:27.576851 6858 consensus_peers.cc:485] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:27.587738 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:27.601537 9148 consensus_peers.cc:485] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:00:27.604112 16980 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:27.646971 6465 consensus_peers.cc:485] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:27.703249 8884 consensus_peers.cc:485] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:27.727407 9148 consensus_peers.cc:485] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:27.776307 8884 consensus_peers.cc:485] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:27.845657 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:27.884617 9015 consensus_peers.cc:485] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:27.897840 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:27.918502 6596 consensus_peers.cc:485] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:27.969293 6465 consensus_peers.cc:485] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:27.990234 6465 consensus_peers.cc:485] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:27.994393 9148 consensus_peers.cc:485] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:28.023666 9148 consensus_peers.cc:485] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:28.042562 6463 consensus_peers.cc:485] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:28.085083 6465 consensus_peers.cc:485] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:28.087577 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:28.128158 16980 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:28.128911 6596 consensus_peers.cc:485] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:28.159117 6594 consensus_peers.cc:485] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:28.168367 8882 consensus_peers.cc:485] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:00:28.181126 17006 raft_consensus.cc:482] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:28.181229 17006 raft_consensus.cc:504] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } } peers { permanent_uuid: "2145d15ab2d74c0287839266f1411a43" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46119 } attrs { promote: false } }
I20240503 14:00:28.181623 17006 leader_election.cc:288] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
W20240503 14:00:28.182463 6596 leader_election.cc:334] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:28.182593 6594 leader_election.cc:334] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:28.182667 6594 leader_election.cc:302] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2145d15ab2d74c0287839266f1411a43; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:28.182804 17006 raft_consensus.cc:2726] T 6ba45113776347c3adeae091c95427df P 2145d15ab2d74c0287839266f1411a43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:28.186965 9015 consensus_peers.cc:485] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:28.215431 6858 consensus_peers.cc:485] T f403dc06195d449e8f454a9f4e795925 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:28.217250 6594 consensus_peers.cc:485] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:28.299208 9017 consensus_peers.cc:485] T 64135eb6ef574c95b3677e32d66748a4 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:28.327308 6596 consensus_peers.cc:485] T 3612cd35d8fa4300a1132fe43a63e6c1 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:28.356040 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:28.363247 8882 consensus_peers.cc:485] T 9f0daf41253f4678b158e3ccbb7b0c98 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:28.378672 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:28.379360 6856 consensus_peers.cc:485] T ea438ff21d9744f697878406493f0fdc P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:28.395823 9017 consensus_peers.cc:485] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:28.402791 8882 consensus_peers.cc:485] T c8b2d8a9ed5f4b09812a85445a8e3b85 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:28.414918 9146 consensus_peers.cc:485] T b7e1060596204c60882388e984986ac6 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:28.453714 6858 consensus_peers.cc:485] T 292ed55b431847f39d832ebe61e6c5e4 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:28.464700 6594 consensus_peers.cc:485] T 4748749ca2e94c83a9cff7e2564ba32a P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:28.471586 9017 consensus_peers.cc:485] T 83aef7ca410b4f9c9aa96a432f290ef4 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:28.489810 6594 consensus_peers.cc:485] T 586369c0c3174070a3e184af38b3adf7 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:28.494177 9146 consensus_peers.cc:485] T 568c2e7206f9496baa22432b72c021b1 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:28.494432 6856 consensus_peers.cc:485] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:28.504544 8882 consensus_peers.cc:485] T 01551845458048e48df9901947b4eec6 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:28.510368 6463 consensus_peers.cc:485] T ce9410a4ed09491b8d37456ae44880a1 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:28.516579 9148 consensus_peers.cc:485] T fc0f85384e434c588213afd4272785c8 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:28.548520 6858 consensus_peers.cc:485] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:28.572902 6856 consensus_peers.cc:485] T b08947e5ddd04e7e8dddfa59ca65f713 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:28.574358 9015 consensus_peers.cc:485] T 96ce36eb7c25413eaae380f32b465df7 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:28.575248 9017 consensus_peers.cc:485] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:28.579521 9015 consensus_peers.cc:485] T 62c77ec99cf443d2b7d48b12c8855883 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:00:28.589111 16993 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:28.589395 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:28.647119 6856 consensus_peers.cc:485] T 7a44f5349cc44be69e9b52f8047d2bb1 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:28.655584 6856 consensus_peers.cc:485] T b5780f4a85b146e2bfbbf7d2feaca316 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:28.662267 9017 consensus_peers.cc:485] T 8482be0fb2784324b77b272e40f57d6b P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:28.668386 9146 consensus_peers.cc:485] T 350ba659df804b95b0692091dc2f68a2 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:28.681092 8882 consensus_peers.cc:485] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:28.696022 6463 consensus_peers.cc:485] T f9d875f4872e4f5da12a326381d523c0 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:28.726428 9146 consensus_peers.cc:485] T 8f3c05440fea43689757d5c5bce8424d P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:28.738229 9148 consensus_peers.cc:485] T a1faecbedbeb4cd685786c1bbd7f5be8 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:28.783371 6858 consensus_peers.cc:485] T dfa48440020d4754be0f5ddf159c23fe P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:28.789319 6463 consensus_peers.cc:485] T 957b89de328d434d8ea03a59bc93fdcb P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:28.812438 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:28.812407 9148 consensus_peers.cc:485] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:28.827107 9015 consensus_peers.cc:485] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:28.841840 9148 consensus_peers.cc:485] T 2ddf3f4cf2884b8785b216a59a5b40f9 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:28.894789 9146 consensus_peers.cc:485] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:28.896801 6858 consensus_peers.cc:485] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:28.913935 9017 consensus_peers.cc:485] T 1ab63490264c4755aeab897f865e9a11 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:28.950372 6858 consensus_peers.cc:485] T d55d270456d3460586ff3cd41787b5d2 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:28.957190 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:28.995100 9017 consensus_peers.cc:485] T ab6fe502a80c410a9b6ee6f5d3d3205e P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.020172 6594 consensus_peers.cc:485] T efe57c7f306b4b0ab934388f5a372a74 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.101812 9017 consensus_peers.cc:485] T ee688e32b7d049509a39259d13439cf6 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.134428 9146 consensus_peers.cc:485] T 37c1f03606fe4e5f8e31a252cce18016 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.138250 6465 consensus_peers.cc:485] T 9a6709e4024d47448c20d33a2b36479e P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:00:29.152043 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:29.155023 6594 consensus_peers.cc:485] T 8d25b0d68b7643cbbbdeb787cac7a1a4 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.155295 9015 consensus_peers.cc:485] T 9253d1943fd74523bafe01c1acf79188 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.164727 6594 consensus_peers.cc:485] T d6af833fca454d0d826c376c95cf1a76 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:00:29.174650 16993 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:29.199868 6594 consensus_peers.cc:485] T 564582aa33a7465081724523096591d3 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.211822 9017 consensus_peers.cc:485] T 3d9906a118744dd6a78d50e52f1961e2 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:29.233747 9146 consensus_peers.cc:485] T 5c7b12e4dc3943a299fc8a49cbc02694 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.246002 6858 consensus_peers.cc:485] T 442c955ab10f4ecbafdc9bf89d8fb39e P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.274220 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:29.283216 6856 consensus_peers.cc:485] T 2f3ac8b5d69745d788bb21a316e0ef21 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:29.312233 6465 consensus_peers.cc:485] T e106b1432baf423493c95fcce63a55da P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.331846 9146 consensus_peers.cc:485] T 133942ceb914456da3f1a281f16d578f P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.339670 8884 consensus_peers.cc:485] T 9af92a2d926543c4aa68a1f85b7da24b P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.341238 9146 consensus_peers.cc:485] T a7e9c80dbf8c41469e54535c20648df4 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.342955 6858 consensus_peers.cc:485] T 6ee85f4483004c1f9499203c7323706d P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.415153 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:29.420399 6858 consensus_peers.cc:485] T 0b397d0460c94438a773be184312b60f P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.425489 8882 consensus_peers.cc:485] T fd45a0cf3a2d4657b02016bc60c5be34 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.485678 6596 consensus_peers.cc:485] T 61dfb1d40ee444368470a0dafc3179a4 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.497401 6594 consensus_peers.cc:485] T 8f57f1ad5c604b5e957374bab029a5df P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.506462 8882 consensus_peers.cc:485] T 5fba96f4e9e14a5eb6fda9ed132f4e08 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.512488 9148 consensus_peers.cc:485] T 0fe53d65ab4d482aad118d6e3bd5533c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.542317 9017 consensus_peers.cc:485] T 44eddd53c3f348a9a1117a20473cbb65 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.555809 6463 consensus_peers.cc:485] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.573170 6856 consensus_peers.cc:485] T d002f7a57be04915b7b0b3dcb985f550 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:00:29.626071 16990 raft_consensus.cc:482] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a85e492c8154e9881a9cc2a105d7e64)
I20240503 14:00:29.626200 16990 raft_consensus.cc:504] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a85e492c8154e9881a9cc2a105d7e64" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46365 } } peers { permanent_uuid: "f58065db70b64ef8b1dc499aa6ae4460" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43531 } } peers { permanent_uuid: "0d77752181e944698e809e325e5937e6" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41931 } attrs { promote: false } }
I20240503 14:00:29.626574 16990 leader_election.cc:288] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365), 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931)
W20240503 14:00:29.627442 6856 leader_election.cc:334] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:29.627849 6858 leader_election.cc:334] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
I20240503 14:00:29.627923 6858 leader_election.cc:302] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58065db70b64ef8b1dc499aa6ae4460; no voters: 0d77752181e944698e809e325e5937e6, 4a85e492c8154e9881a9cc2a105d7e64
I20240503 14:00:29.628068 16990 raft_consensus.cc:2726] T ea45c42d7fed4171a1b726b0684d2fa5 P f58065db70b64ef8b1dc499aa6ae4460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:00:29.634932 6596 consensus_peers.cc:485] T a60d453dd9c04ffaa6af6a6e233c7ef2 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.643888 6856 consensus_peers.cc:485] T af98fddcb623403c841a77ecc10a3e07 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.653592 9148 consensus_peers.cc:485] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.669811 9148 consensus_peers.cc:485] T 1e5875904e6047eb82f28c3cbc9318fc P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.670305 6856 consensus_peers.cc:485] T af8bba02684c4555893def9c1e88ebf0 P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.682253 6856 consensus_peers.cc:485] T 7fde7d6bb07e4dbda5276b0dbbcff19b P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.688187 8884 consensus_peers.cc:485] T e92eaea6e9c8493690b749f162949ac4 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.692593 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
I20240503 14:00:29.780108 16993 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:29.799752 6596 consensus_peers.cc:485] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.806991 6594 consensus_peers.cc:485] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.822980 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:29.865260 8884 consensus_peers.cc:485] T da57af787aef40628e2c043023763f02 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.869184 6596 consensus_peers.cc:485] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.897617 9146 consensus_peers.cc:485] T fdc763e134884bde9e4e66947e5ad126 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.946635 8884 consensus_peers.cc:485] T e29a1f8d9c5c40aaa4023dd3d4204fb5 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.952221 6594 consensus_peers.cc:485] T 00434c9ea1654f7a9a9c31397a71998d P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:29.982093 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:30.079213 9148 consensus_peers.cc:485] T a914c22ceff44fc685aad0fdcd04d844 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:30.083233 9015 consensus_peers.cc:485] T 0a719fd125e54a0fa1c9b42a92548da0 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:30.094182 8884 consensus_peers.cc:485] T ef19b1e5f15e49448c1b2919a24090d3 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:00:30.094498 9148 consensus_peers.cc:485] T 47d38e1d60dd42eabf7191598ded8cd5 P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:30.099483 6465 consensus_peers.cc:485] T 342229593b184019ab915c0389b184b4 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:00:30.187399 16980 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:30.202754 9146 consensus_peers.cc:462] T 4825989d826642bbb5f4bf0ea350e27c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111)
W20240503 14:00:30.217947 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:30.219233 6858 consensus_peers.cc:485] T 10dd65668d544a7fb8d2edc737c303ae P f58065db70b64ef8b1dc499aa6ae4460 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:00:30.335253 8882 consensus_peers.cc:485] T 2ed7d05584d14fdaafb625eaff3ecf08 P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:30.396874 6465 consensus_peers.cc:485] T abe859639139483aa1e57ee6c9722b87 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:30.422132 6596 consensus_peers.cc:485] T 0fa4727297da40eb9f58f5daa8150e0e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:30.423923 6463 consensus_peers.cc:485] T 387d78763d574e57b914e2dbac83a932 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:30.440343 8884 consensus_peers.cc:485] T 174e5b7577d94d87b729d1735987a97a P 06ca8dbc004c42b99faa527a4e6f9545 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:00:30.449723 6596 consensus_peers.cc:485] T e76ded33d780455d963c51a253cbd0e0 P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:30.500854 9017 consensus_peers.cc:485] T 9809e66670ff47f4bb0845304dcea025 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:00:30.528122 6594 consensus_peers.cc:485] T 95fb9ad43cf34a78b98965035cd086ec P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:30.529518 9148 consensus_peers.cc:485] T db13f399b9e04fe9a69e729e384d046c P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:00:30.579996 6465 consensus_peers.cc:485] T a9e87083385d4e6bac9bda95670b68a8 P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:00:30.603155 9017 consensus_peers.cc:462] T aaa93a7fd5df4cb7967d6bdb0c8dd6d1 P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:30.629735 6596 consensus_peers.cc:462] T 21dd1c7418e54d30b665f57ef1842f9e P 2145d15ab2d74c0287839266f1411a43 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111)
W20240503 14:00:30.646054 6465 consensus_peers.cc:485] T aecb348d4b3b4500a587c92a7fefab4e P 5f46e19644fd4f4b825c86ba6af768da -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:00:30.658596 6594 consensus_peers.cc:485] T 752e0d08505b4c8cb4b2a44cc00ce858 P 2145d15ab2d74c0287839266f1411a43 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn't send request to peer 4a85e492c8154e9881a9cc2a105d7e64. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:46365: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:00:30.678306 16980 raft_consensus.cc:1013] T 9ff9b89e4e3247c59ae48ecfeb72f56b P ab1d21935d6049888eb0ceccc3c9d735 [term 1 LEADER]: attempt to promote peer 06ca8dbc004c42b99faa527a4e6f9545: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 14:00:30.696732 9148 consensus_peers.cc:485] T 1eec99c0bb994dd7b6ae63918125bccb P ab1d21935d6049888eb0ceccc3c9d735 -> Peer 0d77752181e944698e809e325e5937e6 (127.0.99.136:41931): Couldn't send request to peer 0d77752181e944698e809e325e5937e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41931: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:30.700629 9015 consensus_peers.cc:485] T 6321c633e38e4c9e853705b2b271326f P 8d574fb4a89d4eea947c859470d44dd4 -> Peer 4a85e492c8154e9881a9cc2a105d7e64 (127.0.99.131:46365): Couldn'